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EXECUTIVE SUMMARY 

Acting as the lead agency for the National STEM Consortium (NSC), Anne Arundel Community 
College (AACC) engaged Hezel Associates to provide an independent program and impact 
evaluation of the U.S. Department of Labor (USDOL)-funded STEM certificate initiative. This 
report is comprehensive and covers the findings from all 4 years of the grant. Findings from the 
first 3 years were previously reported and are summarized and integrated into new findings from 
the final year. 

The 4 years of the mixed method evaluation included a variety of data collection activities, 
including annual interviews with NSC members, advisory board members, employer partners, 
workforce agency partners, and current and past NSC students; and student questionnaires. 
Program documents were also reviewed, as well as extant student data. Data analyses consisted 
of qualitative (using a preordinate scheme) and quantitative (descriptive statistics) methods. 
Findings generated from the analyses provide the basis for conclusions and recommendations for 
program improvement throughout the project. The final yearQ evaluation activities included 
project staff interviews, employer interviews, a student questionnaire, document review, and 
analysis of extant student data. 

High-level evaluation findings for the NSC grant project include the following: 

• Project staff and the management team successfully completed all grant strategies and 
activities outlined in their work plan. 

• Throughout the grant period, the management team provided necessary policy and fiscal 
guidance, as well as frequent and useful communication to the project coordinators, 
making for a well- managed grant. 

• Technical track curricula were completed as a collaborative effort and are publicly 
available online. 

• STEM Bridge has been popular outside of the NSC, with thousands of users from non- 
NSC colleges, high schools, and middles schools. 

• The NSC grant has helped to establish new relationships between the colleges and local 
and regional employers, as well as to strengthen existing relationships. 

• The NSC education model, which included block scheduling, cohort enrollment, 
compressed classroom time, employer linkages, hybrid delivery, one-on-one advising 
with a navigator, and contextualized remediation and refreshers through STEM Bridge, 
appears to have fostered student success in terms of retention and completion. 

• Enrollment targets were met, while the number of completions was slightly lower than 
the target. 

• NSC female participants were less likely than male participants to complete their 
technical track within the expected timeframe. 

• NSC students who completed their programs have been successful in finding relevant 
employment. 

• The staff involved with the NSC have created curricula and implemented an educational 
model that advances student success; however, the future of the consortium is not clear. 
Informal relationships will likely continue, but a formal structure has not been 
established. 
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The following report describes findings from all data collection activities and analysis 
throughout the grant, with an emphasis on previously unreported data from Year 4. Conclusions 
and recommendations for moving forward are also included. 
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INTRODUCTION 

The National STEM Consortium (NSC), a 10-college partnership that spans nine states, began in 
2011 with funding from the U.S. Department of Labor (USDOL) Trade Adjustment Assistance 
Community College and Career Training (TAACCCT) grant program. Led and managed by 
Anne Arundel Community College (AACC), the NSCQ goal is to increase U.S. competitiveness 
in the global economy by creating industry-driven STEM certificate curricula to support local 
and regional labor market needs. Program staff focused on development of 1-year certificate 
programs for the following technical tracks: Composites, Cyber Technology, Electric Vehicle 
Technology, Environmental Technology, and Mechatronics. The 10 NSC colleges are listed in 
Table 1. 


Table 1. NSC Institutions 


Institution 

Location 

AACC 

Arnold, MD 

Cuyahoga Community College (CCC) 

Cleveland, OH 

College of Lake County (CLC) 

Grayslake, IL 

Clover Park Technical College (CPTC) 

Lakewood, WA 

Florida State College at Jacksonville (FSCJ) 

Jacksonville, FL 

Ivy Tech Community College (ITCC) 

Indiana (24 locations statewide) 

Macomb Community College (MCC) 

Warren, Ml 

Northwest Arkansas Community College (NWACC) 

Bentonville, AR 

Roane State Community College (RSCC) 

Harriman, TN 

South Seattle Community College (SSC) 

Seattle, WA 


Over the 4-year grant period, NSC staff developed core curriculum with supporting materials, 
customizable to regional labor markets. Each technical track is intended to be easily adoptable by 
community colleges throughout the United States. Programs were designed for Trade 
Adjustment Assistance (TAA)-eligible workers, but are also accessible to other unemployed and 
underemployed individuals in each collegeQ service area. The certificate programs range from 
two to three semesters and students move through individual programs as part of a cohort. NSC 
colleges employ a variety of support services and job placement assistance for students. In 
particular, most partner colleges have an NSC grant navigator on staff who focuses on studentsO 
program success by connecting them with needed services, as well as employers. A substantial 
component of NSC programming is STEM Bridge, a shared and embedded remediation 
curriculum that unifies the technical tracks. The STEM Bridge curriculum focuses on real-world 
problems in topics such as math and writing skills, and is intended to help participants prepare 
for and succeed in their technical programs. 

The NSC partnered with Hezel Associates in 2012 to provide external evaluation services for the 
grant period. Through a mixed methods approach, Hezel Associates has provided formative 
findings to further project improvement and to assist the NSC in meeting USDOL requirements. 
The following report reflects final evaluation findings and recommendations generated from all 4 
grant years, with a particular emphasis on new data from Year 4. The first 3 years of the 
evaluation were focused on implementation quality. The final year® evaluation was driven by 
the following questions directed at outcomes for both students and NSC colleges themselves, as 
well as scale up opportunities for further work: 
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1. What were the outcomes in terms of the studentsOattainment of certifications, 
certificates, diplomas, or other recognized credentials as a result of the program? 

2. What were the outcomes in terms of student retention rates (i.e., students who 
completed or were still in a program at time of measurement) for TAA-eligible 
workers and other adults participating in the program? 

3. What impact did the NSC programs have on participantsOemployment outcomes? 

4. What impact did the NSC program components have on consortium colleges in 
terms of program and course offerings, student enrollment, and college processes? 

5. What scale up opportunities exist for consortium colleges? 

6. What is the promise of components of the NSC model on community college 
education? 

Questions 1 through 5 are addressed in this report. Question 6 was added near the end of Year 4 
and is addressed in a separate report, entitled NSC Model Report. In addition, a labor market 
analysis was performed for each technical track; those findings comprise a series of NSC labor 
market reports prepared by Hezel Associates. 
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METHODS 

Hezel AssociatesOtheory-driven approach to evaluation emphasizes the linking of project 
objectives, activities, and participant outcomes. The data needed to meet USDOLCS requirements 
and answer questions related to NSC program implementation and impact were collected and 
analyzed using a mixed methods approach, applying strategies specifically aligned with project 
activities and outcomes. 

Instrumentation and Data Collection 

Data collection activities were ongoing throughout the duration of the grant. Hezel Associates 
deployed various instruments in order to assess program implementation and impact. Instruments 
used in Years 2 and 3 are briefly summarized in the following section, since they have 
previously been reported. Year 4 data collection methods are discussed in more detail. 

Document Review 

In Year 4, researchers created a framework for project document review based on the strategies 
and activities outlined in the NSC work plan. Program documents and work products from all 
grant years, including guidance memos and meeting agendas, were made available to the 
evaluation team regularly by NSC staff via a SharePoint system, or shared by the Project 
Director via email. Each document was listed in an index, created by researchers in a spreadsheet 
where information such as document name, document date, document description, and alignment 
with strategy and activity numbers, were recorded. This is included in Appendix A. 

Student Questionnaire 

Hezel Associates researchers developed online questionnaires in Years 3 and 4 aimed at current 
and past NSC students to further explore their perceptions regarding the NSC. The Year 3 
instrument is described in the Year 3 Evaluation Report. In Year 4, the 24-item instrument was 
revised to gather data on studentsOacademic and demographic profile, and employment 
outcomes. Thirteen items focused on demographic information and participant profile data, 
including age, gender, current student status, college attended, and program enrolled. Six items 
related to employment status. Two were focused on the amount of preparation for employment 
students received, where they could rate each item on a 7-point Likert scale ranging from 
strongly disagree to strongly agree. Two questions pertained to the program services and 
components students participated in or used, such as STEM Bridge, internships, or grant 
navigation. One was a yes/no question to indicate use of each service, the other a 7-point Likert 
scale reflecting their satisfaction, ranging from very dissatisfied to very satisfied. All response 
scales had a not applicable option. The Year 4 student questionnaire is included in Appendix A. 

Student contact information was provided to Hezel Associates by each partner college for 
administering the questionnaire, with the exception of CPTC, CCC, and RSCC, so there are no 
data from these institutions. Hezel Associates distributed the questionnaire link via email to 940 
current and former students. Informed consent language was included in the email. One reminder 
was sent to those who had not yet completed the questionnaire. 


Hezel Associates, LLC 


8 


Consortium Member Interviews 

Hezel Associates designed telephone interview protocols for consortium members in Years 1-4 
to help assess the NSC 6; program implementation and potential for sustainability and scale-up. 
Years 1-3 consortium member interviews are described in their respective evaluation reports. 

The Year 4 protocol consisted of eight questions, which focused on institutional changes that 
occurred due to the grant, employment outcomes for their students, and overall opinions on the 
entire project. The interview protocol is included in Appendix A. 

NSC leadership provided contact information for potential interviewees at each college, spanning 
the various NSC technical tracks. Hezel Associates researchers scheduled interviews via email 
and conducted the telephone interviews in February and March 2015. A document explaining 
study details and informed consent was emailed to each staff member who scheduled an 
interview. Completed Year 4 interviews, l ik e previous years, represented all colleges and 
technical tracks. This year, the research team conducted 22 interviews, each lasting 15 to 20 
minutes. Notes were taken by a note-taker during interviews. Interviews were also recorded with 
verbal permission to support the notes. 

Employer/Industry Stakeholder Interviews 

To garner employer and advisory board perspectives on the success of NSC programs and 
emerging industry trends, Hezel Associates created a telephone interview protocol aimed at 
advisory board members, employer partners, and workforce agency partners of the NSC. This 
group was interviewed in Years 2-4. Descriptions of interviews from Years 2 and 3 can be found 
in their respective evaluation reports. The revised protocol for Year 4 consisted of nine 
questions, geared toward each respondentQ relationship with an NSC college, the programsOfit 
into the regional labor market, perspectives on individuals hired from NSC programs, and 
anticipated changes in industry that may affect training needs in the future. The protocol is 
included in Appendix A. 

Representatives from partner colleges provided updated contact information for potential 
interviewees from the previous year. For those colleges that did not provide updated information, 
the previous yearQ list was used. Hezel Associates solicited interviews via email and conducted 
the interviews in March and April 2015. A document explaining study details and informed 
consent was emailed to each participant who scheduled an interview. The evaluation team 
performed interviews with 32 individuals in Year 4, spanning all partner colleges except CPTC, 
and all technical tracks. This is similar to those conducted in Years 2 and 3, where all technical 
tracks were represented. In Year 2, employers associated with all colleges but FSCJ, NWACC, 
and CCC participated, while only CPTC was not a part of the Year 3 interviews, due to lack of 
response from employers. Year 4 interviews lasted 15 to 20 minutes each. Notes were taken by a 
note-taker during interviews. Interviews were also recorded, with verbal permission, in order to 
support the notes. 

Student Interviews 

Student telephone interviews were conducted in Year 3 only and are described in the Year 3 
Evaluation Report. Data from these interviews contribute to the overall grant findings, and either 
support or challenge findings from other data collections. 
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Professional Development Questionnaires 

Questionnaires were administered to attendees of professional development trainings provided 
by NSC faculty in Year 3 only and are described in the Year 3 Evaluation Report. Data from 
these questionnaires contribute to the findings in this report, and either support or challenge 
findings from other data collections. 

Participant Data 

NSC student participant data, such as demographics (e.g., age, gender, race), special status (e.g., 
veteran, TAA-eligible), and program performance (e.g., credits received, completion), were 
made available to Hezel Associates researchers. These data were uploaded into a secure online 
system by each NSC college. NSC colleges were required to submit data quarterly throughout 
the grant period; Hezel Associates accessed and downloaded student data in Year 4 of the grant 
only. Hezel Associates researchers manually aggregated individual-level data across the 
quarterly data submissions, except in some instances where institutions submitted a file of 
cumulated participant data. Due to the process of manually aggregating quarterly data 
submissions, data such as the last date of participation, credits earned, and certificates earned in 
our sample may not reflect what was included in NSC 6, Annual Performance Report to USDOL, 
as Hezel Associates were not provided with the most recent or most accurate data submissions 
from each college. In addition, Hezel Associates excluded data from FSCJ submitted in January 
2015 due to missing data required to aggregate files, select cases from AACC and NWACC due 
to unclear or missing data, and all data from ITCC due to improper data submissions. 

Data regarding participant employment and wages were deemed too inconsistent to perform an 
analysis by the NSC, the Jacob France Institute (who was contracted by the NSC to collect most 
of the employment and wage data), and Hezel Associates. Because the consortium is widespread 
geographically, wage data needed to be collected from nine states. These data were inconsistent 
in their reporting times; therefore, the lag time between a participants completion date and when 
his or her wage records were available was typically several quarters. Additionally, analysis with 
a comparison group was not possible, as most institutions do not collect social security numbers 
of non-TAACCCT participants, which is a necessary identifier for state wage records. 

Data Analysis 

Analysis of data pertaining to the NSC evaluation consisted of a variety of qualitative and 
quantitative methods, as detailed in the following section. Data from each collection source were 
analyzed separately, and then compared for consistent or conflicting findings. Only analysis 
methods for Year 4 data are discussed here. 

Document Review 

Documents housed on SharePoint or sent via email were reviewed against the document review 
framework. Each document was examined and its alignment with a strategy and/or activity was 
noted. After review of all documents, judgements were made for each strategy and activity, 
regarding whether documentary evidence indicated it was met and if so, completed in the time 
period stated in the NSC work plan. Where appropriate, evidence was further broken down by 
technical track. 
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Student Questionnaire 

Frequencies were calculated for demographic, student profile, marketing, and employment data, 
and for the Likert- scale items. Cross tabulations were performed to determine patterns between 
program of enrollment and prior education level, as well as between program enrollment and 
overall experience. 

Interviews 

Researchers used a preordinate scheme to guide qualitative analysis of the data collected using 
the two interview protocols (Employer/Industry Stakeholder Interviews and Consortium 
Members Interviews). Through this approach, lengthy discussions were parsed into bits of 
content, which were then fitted to the conceptual framework established by the evaluation topics. 
Each excerpted bit was tested against not only the construct of interest, but also the accumulating 
narrative content associated with it, applying a condensed constant comparative method to 
isolate each construct and clarify how it was labeled or coded (Dey, 1993). Researchers then 
identified logical linkages among the named constructs. These patterns became themes that 
explained the semantic relationships among grant activities and outcomes for participants. 

Participant Data 

Hezel Associates researchers analyzed extant data from nine NSC schools. Analysis consisted of 
frequencies of outcome measures by school, by technical track, and in the aggregate. Researchers 
cross-tabulated the number of completed certificates and credentials with variables including 
college, technical track, TAA eligibility, age, gender, and ethnicity. Researchers also created 
variables to determine the number of students who completed a certificate or credential, did not 
complete and withdrew, and did not complete and were still enrolled. These data were then 
cross-tabulated with variables such as TAA eligibility, Priority of Service status (veterans and 
spouses of veterans who take precedence over non-covered individuals in USDOL-funded 
employment and training 1 ), gender, ethnicity, and developmental need (e.g., math, reading, or 
English). Lastly, Hezel Associates researchers created variables for students who completed their 
certificate or credential within the appropriate amount of time for their certificate program and 
those who took longer. These data were cross-tabulated by TAA eligibility. Priority of Service 
status, and developmental need. 


1 https://veterans.workforce3one.org/page/priority-of-service 
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FINDINGS 

The following describes the findings for the entire 4-year grant period. These are separated by 
implementation, student outcomes, NSC college outcomes, and sustainability opportunities. 

Program Implementation 

The first 3 years of the NSC evaluation focused largely on implementation, such as the 
consortium organizational structure and grant management, and the successes and challenges of 
putting the NSC model in place at the member institutions. The following section outlines 
implementation of the NSC model as it was intended, followed by a detailed description of how 
the project was actually implemented. 

Intended Implementation 

The overarching goal of the NSC was to create programs that will help to improve retention and 
completion rates among consortium schoolsOstudents, as well as to reduce time to completion. 

As laid out in the NSC technical proposal to USDOL, the project intended to design and 
implement 1-year certificate programs in five technical areas: Composites, Cyber Technology, 
Electric Vehicle Technology, Environmental Technology, and Mechatronics. The programs were 
chosen based on labor market demands in the regions surrounding the 10 consortium colleges. 
Within each technical track are structural elements that had prior evidence of success in the 
literature. These include: block scheduling; cohort enrollment; compressed classroom time; 
embedded student support; employer involvement; hybrid delivery (face-to-face and online); and 
an accelerated bridge program called STEM Bridge, which provides student remediation in math, 
reading and writing, and soft skills. 

The grant team emphasized in the technical proposal that the elements described above provide 
the most impact when they are implemented together. In other words, no one element will ensure 
student success on its own. The length of the programs, averaging 30 credit hours; block 
scheduling; compressed classroom time; and hybrid delivery were all intended to reduce time to 
completion, while cohort enrollment; embedded student support; employer linkages; and STEM 
Bridge were intended to ensure students were best equipped to gain skills, complete their 
certificate, and find employment. Specifically, STEM Bridge was to be consistent across all 
tracks and colleges, and included in each. It was intended to forgo traditional developmental 
education and allow adult learners to refresh their math, reading and writing, and critical thinking 
skills as a part of their certificate program. 

The curriculum itself, including STEM Bridge, was intended to be high quality, integrating 
employer input and state-of-the-art equipment for training. Once developed, the final curriculum 
materials were to be made public through an Open Educational Resource (OER), where other 
colleges could easily find and use individual materials, courses, or entire technical tracks. 

Implementation Summary 

The actual implementation of the NSC model at the 10 colleges was very close to what was 
intended, with some deviations, as described in the following paragraphs. 

Management. The grant was managed by the team at AACC, where they coordinated high-level 
grant activities, facilitated deliverable completion, and established financial procedures to ensure 
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USDOL compliance. The management team consisted of the Project Director (from AACC); the 
Principal Investigator (also from AACC); a representative from AACCQs Office of Sponsored 
Programs, who guided the financial aspects of the grant; the curriculum development team, 
created to lead the technical track course development; the Learning Outcomes Specialist (a 
consultant); the evaluator (consultant); and other team leads as necessary. The management team 
held weekly conference calls throughout the course of the grant, facilitated by the Project 
Director. The topics of these calls were specific to the grant stage and were intended to provide 
status updates to all members and to work through any issues needing attention. Topics included 
curriculum development, staff hiring and contracting, grant policies and procedures, and any 
other pertinent items. 

In terms of communication with all NSC members, the management team employed a variety of 
methods. A file sharing system was established on SharePoint, where all procedural documents, 
technical team information, annual and quarterly USDOL reports, and meeting notes were 
housed. Guidance memos were issued to all grant staff for topics that were of high importance. 
Over 30 such memos were developed during the grant, covering items such as budgeting 
instructions, reporting templates, outreach material creation guidelines, and curriculum 
development direction. These served as a formal set of procedures that helped to ensure 
consistency across the 10 colleges. The team also held bi-weekly conference calls with project 
coordinators from all 10 colleges and the 5 pathways throughout the course of the grant. These 
had pre-set agendas and covered topics pertinent to the grant stage, such as curriculum 
development and student enrollment. In each grant year, an NSC Summit was held in Annapolis, 
MD. These brought together representatives from all 10 colleges for 2-3 days, allowing them to 
interact face-to-face and share best practices and ideas. These were also a forum for making 
progress on action items, such as curriculum development, where technical teams met and 
worked through issues including sticking points, which are additional instruction designed to 
guide students through particularly difficult content components, and teaching toolkits, which are 
course-level guides. 

One of the main undertakings the management team handled over the course of the grant was the 
request for and administration of a 1-year, no-cost extension. Because the Round 1 TAACCCT 
grants were 3-year awards (the subsequent rounds are 4-year), the NSC, like other Round 1 
awardees, found there was not enough time to develop and implement curriculum to serve their 
targeted number of students. In the request, the NSC lowered its enrollment goal by 15% (from 
1,506 to 1,279), as a lower target was expected to be a more realistic goal. The award of the 
extension in March 2014 entailed a great amount of budget and project activity restructuring on 
the part of the management team and the project coordinators at each college, however, has 
allowed the NSC to meet its enrollment goals. 

In terms of the project work plan, the start-up of coordination, policies, and procedures of the 
NSC (Activity 1.1) were all completed, with some completed slightly behind schedule (expected 
to be complete by February 2012; mostly complete by September 2012), however, all were 
started when intended. In Year 1, interviews with project staff revealed that they viewed the 
grant start-up as a slow and informal process. Some of the delays in Activity 1.1 completion 
were due in part to grant management staffing changes, such as the shift of Principal Investigator 
and the departure of the initial Project Director in May 2012. However, for a project of this size 
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and complexity, a slight delay in these activities is not unexpected and in this instance did not 
appear to negatively affect subsequent project work. 

Establishment and revision of NSC policies (Activity 1.1, Deliverable 1) have been ongoing; 
however, start-up policies were officially in place only a few months behind schedule. 
Deliverable 2, implementation of fiscal procedures, was completed according to the timeline. 
Hiring of staff and technical consultants (Deliverable 3) tended to fall behind schedule, 
particularly for Subject Matter Experts (SMEs) to review curriculum. However, many of these 
individuals were not hired until needed, which was in practice after the anticipated end date of 
February 2012. The bulk of the management staff and consultants needed for the grant start were 
hired on time. Deliverable 4, establishment and meeting of steering committees, work teams, and 
cross-cutting teams, was completed over a year later than anticipated. This is understandable, 
considering the additional time needed to hire staff at each college. 

Activity 1.6 is focused on the ongoing formation of policies and procedures to foster grant 
success and the ability to sustain and scale grant deliverables. This was pursued for the entire 
grant period, as was intended. The last two years of the grant showed substantial activity 
surrounding sustainability and scale-up. A draft sustainability plan was created in early 2013, 
and has been continually refined and updated due to new research and new partnerships. Of note, 
a Leadership Summit was organized by grant management and held in Baltimore, MD in 
November 2014. The objective was to bring together leadership from each NSC college, such as 
department deans, college presidents and vice presidents, along with other high-level staff, to 
learn about the NSC model and its outcomes, and to brainstorm ways to sustain the work 
completed and the partnerships established. A11 colleges were represented at the meeting and the 
team was able to make progress in identifying project elements that are desirable to sustain. A 
team lead meeting was convened shortly after the Leadership Summit in Annapolis, MD, where 
the technical team coordinators further shared and agreed upon sustainability elements. And, the 
final NSC Summit, held in April 2015, focused largely on sustainability of the NSC model and 
what relationships and partnerships will look like in the future. This was attended by the grant 
management team, team leads, grant navigators, and other members of the consortium from most 
colleges. In general, sustainability and scale up of the NSC has been a major focus over the last 
two years, and data were collected to inform these areas as a part of the Year 4 evaluation. The 
findings are discussed in the Opportunities for Sustainability and Scale Up section. 

The general consensus among consortium members was that the grant was managed well. In 
interviews throughout the grant period, the management team was praised for being organized, 
keeping project staff on track with deliverables, and maintaining frequent and effective 
communication amongst members. 

Curriculum. A11 five technical tracks were developed and implemented during the grant period, 
satisfying Activity 1.3, Deliverable 1. Table 2 on the following page illustrates which colleges 
participated in development for each technical track. RSCC assisted in the Composites 
curriculum development, but did not fully implement the program due to a decline in expected 
local industry demand. A11 other programs were implemented. Development fell behind schedule 
(slated to end in late 2012; actually completed in late 2013); however, additional staff hiring time 
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likely contributed to the delays. Most pathways were launched in spring of 2013. Curricula 
integrated interactive elements, including online modules and sticking points. 

Curricula generally contained consistent core content within each track, as stipulated in Activity 
1.3, with some track-specific credits embedded due to regional industry demand. For example, 
within Electric Vehicle Technology, one college trains for electric vehicle development, while 
another concentrates on service technician training; therefore, their curricula differ. The 
Mechatronics programs are relatively consistent; however, equipment used differs based on what 
employers use in a particular region. The Cyber Technology programs are very similar across 
colleges because they are focused on specific national certification exams. 


Table 2. Partner College Curriculum Development Participation 


Composites 

Cyber Technology 

Electric Vehicle 
Technology 

Environmental 

Technology 

Mechatronics 

LEAD: SSCC 

LEAD: AACC 

LEAD: MCC 

LEAD: FSCJ 

LEAD: CLC 

CPTC 

CCC 

ITCC 

CLC 

AACC 

CCC 

FSCJ 


ITCC 

CCC 

RSCC 

ITCC 


NWACC 

FSCJ 


NWACC 



ITCC 





RSCC 


The curriculum development process was a collaborative effort. Each track was guided by a 
technical team, facilitated by members of the lead institution (as indicated in Table 2). Teams 
mainly worked together virtually, meeting in-person when able. Curriculum development 
requirements and a schedule was put in place in July 2012 for Phase I of the process, via a 
guidance memo. This stipulated that: (a) at least one employer must review curriculum for 
industry relevance for each technical track, (b) each college must have the curricula approved by 
their own college and state higher education system, and (c) the NSC management team must 
approve curriculum. SMEs were involved throughout development, and helped to guide the 
process to create the content. A Learning Outcomes Specialist was also contracted to inform the 
instructional aspects of the design. A second set of SMEs were engaged in the final review 
process (Phase II), where they ensured that curricula met industry standards and skill needs, and 
had the ability to be disseminated to other colleges. This review was also designed to foster a 
smooth transition into OER. 

This process was touted as a strength of the NSC grant, as it was a streamlined method of 
creating new materials across institutions and regions, engaged SMEs and employer input, and 
was intended to be nationally portable from the beginning. It consisted of utilizing various 
templates for materials and required a high level of formatting consistency. While this was a 
necessary feature of the NSC model, some project staff conveyed that inconsistencies of 
curriculum format expectations at the beginning of the project caused setbacks and delays in 
material completion. However, these issues were resolved and curricula have been finalized. 

A major aspect of the NSC is to provide content that can be easily adopted and used outside of 
the grant program. Activity 1.3, Deliverable 3, creation of guides for faculty and staff, was built 
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into the grant to address this. Teaching toolkits were established, which provide course-level 
details for instructors to use. A (program guided for each technical track was also created, which 
outlines essential elements of implementation, such as equipment needs and course materials. 
While initially intended to be completed by late 2012, this deliverable was better suited to be 
completed in conjunction with the OER conversion, later in the grant. The no-cost extension also 
allowed this work to be extended. 

As a way to ensure the curricula were delivered as intended and with quality, professional 
development was provided for instructors, which focused on content knowledge associated with 
respective technical tracks (Activity 1.3, Deliverable 4). Some experienced course instructors 
visited other consortium colleges to collaborate and share information on the subject matter area, 
while some provided online training in the form of webinars. Informal questionnaires 
administered to professional development attendees by Hezel Associates in Year 3 showed that 
trainings were well received and useful. However, there were several types of trainings 
throughout the grant period, and these surveys provided only a limited view into their quality. 

Lastly, all technical tracks and STEM Bridge curricula are required to be nationally portable 
(Activity 1.3, Deliverable 1) and available as OER. To accomplish this, the NSC partnered with 
Carnegie Mellon University® Open Learning Initiative (OLI), which provides a platform for free 
and accessible online courses. The curriculum and technical teams worked closely with OLI 
throughout the grant period to ensure formatting of materials were consistent with the online 
platform. The Cyber Technology curriculum was the first technical track to be made available on 
OLI, in April 2015. USDOL has also provided an online repository to house TAACCCT-created 
curricula and related support documents, called Skills Commons. As of the end of the grant, all 
technical tracks, composed of 68 courses and STEM Bridge modules, are available on OLI 2 and 
Skills Commons 3 for free use. 

STEM Bridge. A major work effort of NSC was the creation of STEM Bridge, developed as 
online content for adult learners. The development process began early in the grant period and 
was completed when intended. The team, led by staff at AACC, developed STEM Readiness and 
STEM Foundations, both currently available on the OLI and Skills Commons website. STEM 
Readiness is a set of online modules that are intended to help a student refresh his or her skills in 
math (arithmetic, algebra, and geometry), communication, and professionalism. Real world 
scenarios are built in, developed using input from employer partners, covering topics such as 
email communications, customer service skills, and troubleshooting. STEM Foundations was 
designed to help develop studentsOwriting, reading comprehension, and overall communication 
skills. 

The first module of STEM Readiness was implemented in January 2013. The full STEM Bridge 
package was implemented by module over 2013 and 2014, and was completed on OLI in 
November 2014, fully satisfying expectations set for Activity 2.1. While STEM Bridge was 
completed on the expected timeline, some project staff commented that early cohorts began 
before it was in place, putting them at a disadvantage. Implementation of STEM Bridge varied 
between the colleges and technical tracks, as most allowed students to take modules throughout 


2 http://oli.cmu.edu/teach-with-oli/review-our-free-open-courses/ 

3 https://www.skillscommons.org/ 
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their program and others required that they completed it before their program began. Some 
colleges required students to complete some modules as part of their program, while others 
allowed students to only take it if they felt they needed the additional help. 

Activity 2.2 was focused on STEM Bridge quality and usability. NSC technical teams and 
several employers were a part of the development process, ensuring the technical content was 
accurate and reflected industry needs. The Learning Outcomes Specialist was tasked with 
externally validating the modules, verifying that the content aligned with the NSC technical 
tracksOlearning objectives. Once STEM Bridge was launched on OLI, NSC technical teams and 
industry continued to vet the content, alerting the project team to errors or providing 
recommendations for improvement. These changes have been incorporated when deemed 
necessary. This process satisfies Deliverable 1, as it was completed within the anticipated 
timeframe. 

Data are continually collected and analyzed by project staff and the evaluator to examine the use 
of STEM Bridge (e.g., how many used it, who used it) and to determine if there were any 
necessary mid-course corrections that would make it more effective or user-friendly. Usage data 
reported to project staff from OLI showed that STEM Bridge has had over 11,000 users since its 
launch (more than 80% of these were STEM Readiness). According to project staff, only about 
12% of the users were NSC participants, the rest were from other colleges, high schools, and 
even middle schools. 

Additionally, in each grant year the evaluation included a review of perceptions relating to 
STEM Bridge among students and project staff. Staff overwhelmingly found the content to be 
helpful to students and touted the importance of its flexibility, which allowed them to take the 
coursework at their own pace. Students were also satisfied with the self-pace of STEM Bridge. 
However, early in the grant, some expressed frustration that they were required to complete the 
modules as part of their certificate program, when they did not need it. Several stated that it was 
too easy. In Year 3, less than half of the students surveyed found it beneficial. Year 4 student 
questionnaire data indicate a higher level of satisfaction with STEM Bridge than previous years 
(68% of respondents were satisfied), which may be due to the fact that only those who needed it 
were required to complete it. The annual evaluation reports, in addition to user analytics reports 
from OLI, satisfy Deliverable 2. 

Finally, STEM Bridge was intended to be nationally disseminated (Deliverable 3). The entire 
curriculum is currently housed on OLI and Skills Commons for free public access. Project staff 
have presented about STEM Bridge at more than 10 national conferences, including the League 
Learning Summit and the League for Innovations conference, and there appears to be great 
interest in it among post- secondary and secondary level educators. Usage of the content is 
continually increasing. Deliverable 3 has been successfully met. 

Model Elements. Other components of the NSC model include a navigator position at each 
college, cohort enrollment, block scheduling, compressed classroom time, hybrid course 
delivery, and employer linkages. These elements were implemented somewhat consistently 
between the 10 colleges. The program delivery methods and student support services fall under 
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Activity 1.3, Deliverable 2, while employer linkages is a focus of Activity 1.2. These activities 
and deliverables were met on the anticipated timeline. 

The navigator position has been touted as one of the most effective aspects of the NSC grant, in 
terms of recruiting and retaining students. A navigator (in some cases, more than one) was hired 
at most colleges to work with NSC students one-on-one to ensure program success, either as a 
full-time or part-time position. Their role typically consisted of enrollment and admissions 
assistance, guidance through the program requirements, and job placement assistance. This role 
generally went much further, however, as navigators were often the first point of contact for 
students who encountered problems, whether in their coursework or in their personal lives. 
Navigators were well- versed in external supports and often referred students to services within 
the college or outside of the college. They essentially guided students through the nwhole life 
cycled of the program, and by getting to know them on a personal level, were able to give 
tailored support. In addition, a substantial portion of their time was spent on facilitating 
connections between employers and the students. Navigators maintained existing relationships or 
established new associations with local employers. This, along with their familiarity with 
studentsOabilities, allowed them to facilitate appropriate job placements. This process was 
typically so successful, several employers reported that they would approach the navigator for 
new hires before posting an open position publicly. Students also expressed positive opinions, as 
88% of those surveyed who interacted with a navigator were satisfied with the results. The 
colleges who were not able to hire a navigator or those who hired someone late in the grant 
period noted that their students would have benefitted more if one had been in place. 

Cohort enrollment was also a unique feature of the NSC model, with groups of students entering 
a technical track together, attending the same courses, working through curriculum at the same 
time and pace, and completing together. Most colleges implemented this structure, enrolling 
groups of 15-25 students together. Program staff commented that the cohort structure is better 
than the traditional approaches, because it (a) builds camaraderie between students, (b) makes it 
easier to track student progress, (c) creates an anticipated timeline for employers to hire new 
completers, and (d) forms an environment that encourages teamwork, much like the workplace. 
That said, some program staff cautioned that the cohort model does not work well when there are 
too few students, as any personality conflicts among students become amplified. Of students who 
responded to the Year 4 questionnaire, 91% indicated that they were satisfied with the cohort 
model. 

Several colleges implemented block scheduling for the NSC technical tracks, where students had 
fewer and longer classes each day compared to the traditional shorter classes a few times per 
week (often four hour blocks, four days per week). Some were not able to use this structure, as it 
was not compatible with the overall registration and scheduling system of the college. Almost 
two-thirds of students surveyed who were in block scheduled courses were satisfied with the 
structure. Related to block scheduling, the model emphasized compressed classroom time, 
defined as completion of the same number of contact hours as a traditional college program 
schedule, but delivered over a shorter number of weeks. Most technical tracks were 
approximately 30 credit hours, completed over the course of one year (some with summer 
enrollment or an internship component). The composite technical track was designed as a 6- 
month program. Most program staff identified the compact timeframe as a positive element, as it 


Hezel Associates, LLC 


18 



(Worked smoothlyOfor some tracks, l ik e Mechatronics. The short time to completion is also 
beneficial for students and employers, as the time between the start of training and employability 
occurs quickly. About 75% of students surveyed in Year 4 were satisfied with this timeframe. 
However, the condensed format may not have been ideal for the Cyber Technology program, as 
both program staff and students indicated that there was simply too much material to cover. This 
caused some students to be unable to complete industry certifications before completing the 
program, which are crucial for employment in that industry. 

NSC technical tracks were intended to be a mix of in-person and online instruction. All tracks 
implemented this to some extent, either with individual courses incorporating a portion of online 
instruction, or one or two fully online courses embedded in the program. However, program staff 
stressed that online instruction is not ideal for all technical courses, particularly in the 
Composites and Electric Vehicle Technology tracks, because they require hands-on training in a 
lab setting. In addition, some embedded industry certifications cannot be completed online, 
including the OSHA and HAZWOPER training in the Environmental Technology track. Students 
interviewed in Year 3 also underscored the importance of hands-on training, with many of them 
preferring it over online coursework. That said, online courses or components of courses were 
integrated throughout NSC tracks as appropriate. More than 65% of students surveyed in Year 4 
were satisfied with the mix of online and in-person instruction. 

Partnerships with higher education leaders, employers, and workforce development agencies 
(Activity 1.2) have been a highlight of the NSC grant. Each college advisory board consists of 
college leaders, such as deans and department heads. The management team is also composed of 
leadership from AACC. The Leadership Summit described previously brought together high- 
level college leaders and reinforced partnerships among the colleges. The NSC partnered with 
the Wadhwani Foundation midway through the grant period in order to further advance the 
Cyber Technology track curriculum. NSC management has also had several interactions with the 
Gates Foundation, by presenting at their postsecondary education meetings and inviting 
representatives to attend the Leadership Summit. 

Each college and technical track has several companies who serve a variety of roles: advisory 
board members, curriculum reviewers, guest lecturers, liaisons between the colleges and industry 
associations, and providers of faculty professional development (Activity 1.4, Deliverable 2). 
They have also donated equipment for training and participated in mock interviews to help 
students prepare for real interviews. Many also worked with the respective college and technical 
track to develop internships for NSC students. As discussed in the Employment Outcomes 
section, many of these internships have resulted in permanent positions for NSC program 
completers. Consistently throughout the grant period, employers associated with NSC have been 
pleased with the programs in terms of content and structure, the relationships they have formed 
or strengthened with the colleges, and the program completers they have hired. 

The NSC colleges all have a relationship with a local or regional workforce agency (Activity 1.4, 
Deliverable 3); however, the extent of their involvement in the NSC programs varies. Those that 
are highly involved assist with recruitment by referring new students to the technical tracks and 
tracking employment of completers. Several program staff members acknowledged that since the 
grant began, they have seen their relationship with the local workforce agency expand and 
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improve, particularly as representatives become more familiar with the NSC programs. While 
most reported positive partnerships, a few program staff have seen their workforce agencies 
involvement decrease in the last few years; however, they noted that this was mostly due to 
agency organizational changes. 

Student Outreach and Enrollment. Recruitment strategies were a large part of project 
implementation across all colleges, and are the goal of Activity 1.4, Deliverable 1. Early in the 
grant period, an NSC website 4 was established, which gives an overview of the project, the 
technical tracks, and contact information for each college and track. An overarching NSC 
outreach plan, offering general guidance on outreach approaches and templates for physical 
materials, was put in place in early 2014. Each technical track developed their own marketing 
materials to recruit students, with guidance from NSC management. These included flyers, 
brochures, posters, email blasts, and social media posts. Each track also developed promotional 
videos, which featured success stories of students who were in or completed a pathway. These 
were made available in the NSC website, as well as the individual college websites. Colleges 
also collaborated with their local workforce agencies to reach potential students. Students 
surveyed in Year 3 identified the workforce agencies or unemployment offices as the most 
common way they first heard about their program. 

Student enrollment (Activity 1.4, Deliverable 4; Activity 1.5) officially began in January 2013 
and has continued to date. As of June 30, 2015, 1,372 students enrolled in an NSC technical 
track. This exceeds the goal established in the no-cost extension contract, of 1,279 participants. 

A high level of student persistence is another important outcome in the NSC programs, as simply 
enrolling is not a guarantee that a student will receive the full benefits of the program. Over the 
course of the grant period, the term-to-term persistence rate was an average of 82%, which 
exceeded the target of 80%. This also exceeds the national average for community college term- 
to-term persistence (75%; National Student Clearinghouse, 2011). The completion rate of 
participants across all NSC institutions was an average of 69% over the grant period (the national 
average is 21%; National Center for Education Statistics, 2014), and 656 participants have 
completed their program and earned a credential as of June 30, 2015, 64% of the revised no-cost 
extension project goal of 1,023. The total completions at grant end will likely be higher than 656; 
however, these data are not yet available. It is not clear if the completion goal was met by grant 
end. 

The ultimate goal of the NSC is for students to gain relevant employment. While actual data are 
not available due to a lag time in state employment records, NSC staff have estimated the overall 
average of completers gaining employment at 70%. The Employment Outcomes section of this 
report provides more detail on these data. 

According to the reported data, the NSC has fulfilled its enrollment goals, has shown students to 
persist at a much higher rate than the national average, and came close to its completion goals. 

Data. In order to accurately communicate the outcomes of the NSC project, data are continually 
collected and distributed (Activity 1.7). The first deliverable under this activity, reporting to the 
USDOL, has been met, as the NSC has submitted quarterly and annual reports consistently 


4 http://www.nationalstem.org/ 
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throughout the grant period. Deliverable 2 stipulates the sharing of evaluation findings with 
stakeholders. Hezel Associates, the third party evaluator, has provided the NSC with annual 
reports, as well as more rapid turnaround summaries of various data collections. Annual 
evaluation reports were uploaded to the NSC SharePoint site and distributed via email to all NSC 
team members. Data summaries were shared with the NSC team via email as well. Evaluators 
have briefly summarized findings of each annual report and data summary during grant-wide bi- 
weekly conference calls. 

Lastly, distribution of sustainability plans is the focus of Activity 1.7, Deliverable 3. As 
discussed previously, a plan was developed and shared with NSC members in 2013. This plan 
has been continually updated and re-distributed to the NSC team. 

Student Outcomes 

Attainment of certificates and retention rates of NSC students were explored in a quantitative 
analysis. These data were further broken down to look at completion and retention by various 
demographics, including age, gender, and Priority of Service status. The following section 
highlights the findings. Of note, SSC, MCC, and CPTC had the highest percentage of completers 
overall. FSCJ and NWACC had a lower percentage of completers overall, which may be related 
to the low percentage of completers in the Environmental Technology track offered at both 
colleges. In addition, most NSC participants completed their programs within the appropriate 
timeframe. 

Certificate Attainment Outcomes 

One of the primary goals for NSC student outcomes is the attainment of academic certificates 
and industry certifications. Data used in descriptive calculations in this section are based on the 
data available to Hezel Associates and do not represent the most accurate or up-to-date data from 
each college. 

The percentage of students who attained an academic certificate or industry certification varied 
by college, as denoted in Figure 1. These percentages were calculated based on the total number 
of students attaining a certificate divided by the total number of participants at each college. 
While all participants are included in the denominator, some of these students are still enrolled in 
the program and could not have earned the certificate at the time this analysis was completed. 
Therefore, the final percentages may be higher once all participants complete their enrollment 
period. As of this study, SSC, MCC, and CPTC students achieved a completion rate above 85%. 
However, less than 25% of NSC participants from CLC, NWACC, and FSCJ completed their 
program of study. The three schools displaying the highest completion rates solely focus on one 
technical track, while the three schools with the lowest completion percentages are developing 
multiple technical tracks. The three colleges with the highest completion rates also had fewer 
participants enrolled than some of the other colleges, including AACC, CCC, and FSJC. 
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Figure 1. Percentage of NSC Participants who Completed by College 

Figure 2 presents the percentage of completers by technical track. The Electric Vehicle 
Technology track was implemented at two colleges and produced the highest percentage of 
completers, at 88%. The Composites program, available at three colleges, was completed by 
67% of students. Only 9% of students enrolled in the Environmental Technology track 
completed a certification. Both the Mechatronics and Cyber Technology tracks produced a 
substantial number of certified completers; however, their completion rates were 40% and 35%, 
respectively. 
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Figure 2. Percentage of NSC Participants who Completed by Technical Track 

Nine TAA-eligible participants were enrolled in NSC programs, according to the available data. 
Note that more than nine TAA-eligible participants enrolled; however, data were only available 
for the nine examined here. Five students completed a certification as shown in Figure 3. FSCJ 
enrolled three TAA-eligible students, however, none of those NSC participants completed a 
certification. 
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Figure 3. Percentage of TAA-eligible NSC Participants who Completed by College 


Figure 4 displays the percentage of TAA-eligible participants who completed a certification for 
each technical track. The Mechatronics program had the most TAA students enrolled, however 
only 40% attained a certification. No TAA-eligible participants enrolled in the Environmental 
Technology track. 
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Figure 4. Percentage of TAA-eligible NSC Participants who Completed by Technical 
Track 


SSC, MCC, and CPTC achieved a completion rate above 85% for participants who qualified for 
Priority of Service according to the USDOL (e.g., veterans, active duty service members, reserve 
members, and spouses) enrolled in NSC programs (Figure 5). Not surprisingly, those three 
schools also have the highest rates of overall participant completion. CLC, NWACC, and FSCJ 
had the lowest percentage of veteran completers, and constitute the three lowest percentages of 
overall completion rate as well. None of the eight veterans enrolled in an NSC program at CFC 
attained a certification, while all Priority of Service participants at MCC completed their 
program of study. FSCJ enrolled the highest number of veterans (56), but had the second lowest 
completion rate (21%). 


Hezel Associates, FFC 


23 









MCC (n = 10) 


100 % 


SSC (n = 15) 
CPTC (n = 14) 
RSCC (n = 7) 
MCC (n = 27) 
CCC (n = 27) 
NWACC (n = 8) 
FSCJ (n = 56) 


CLC (n = 8) 0 % 


Figure 5. Percentage of Priority of Service NSC Participants who Completed by 
College 

Figure 6 shows the percentage of Priority of Service participants completing a certification by 
technical track. The Composites program had 74% of participants complete a certificate. All of 
the Priority of Service enrollees in the Electric Vehicle Technology track completed their 
program of study, while none of the 16 enrollees in the Environmental Technology track 
completed a certification. The Cyber Technology (48%) and Mechatronics (38%) tracks 
represent similar percentages of completion for Priority of Service participants when compared 
to completion rates for all students. 
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Figure 6. Percentage of Priority of Service NSC Participants who Completed by 
Technical Track 

The mean age for program completers across all schools ranged from 33 at CPTC to 42 at MCC. 
The median age, as indicated in the box and whisker plots in Figure 7, were consistent with the 
means. AACC and SSC had the oldest participants at 71 years old, while CLC had the youngest 
at 18. Overall, the ages of NSC completers were similar across all colleges. 
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Figure 7. Age of Completers by College 


The mean age of completers by technical track ranged from 34 in Environmental Technology to 
42 in Electric Vehicle Technology. The mean ages were similar to the median ages indicated in 
Figure 8. The Environmental Technology track had the smallest range in age of completers, 24 to 
47 years old, which may be due to the small number of participants who completed. 
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Figure 8. Age of Completers by Technical Track 


Completion rates by gender are displayed in Figure 9 for each NSC institution. All of the six 
female participants at MCC completed their NSC technical track. Similar to overall completion 
rates, a sharp decline exists in the percent of male and female certification attainment after the 
top three colleges: SSC, CPTC, and MCC. No female students completed a certification at either 
NWACC or FSCJ. MCC and CCC are the only NSC colleges where a higher percentage of 
female students completed a certification than male students. 
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Figure 9. Percentage of Female and Male Participants who Completed by College 


Figure 10 denotes the percentage of male and female students who attained a certification for 
each technical track. The Electric Vehicle Technology program saw 100% of female students 
complete their certificate compared to 86% of male students. Female students in Composites and 
Electric Vehicle Technology tracks completed certificates at a higher rate than male students; 
however, similar or only slightly lower percentages of female participants completed 
certifications in the remaining three technical tracks. 
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Figure 10, Percentage of Female and Male Participants who Completed by Technical 
Track 

The percentage of NSC participants by ethnicity differed at each college, as displayed in Table 3. 
Overall, white students attained the most certifications, but was also the ethnicity with the 
highest enrollments. The one exception was CCC where 57 African-American students 
completed an NSC program compared to 34 white students. However, the completion percentage 
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for African-American students was highest (97%) at SSC compared to at CCC (40%). Four 
white students completed a certification at NWACC, zero Hispanic students completed a 
certificate, and no other ethnicities were represented in the enrollment. SSC consisted of students 
who completed a certification from all six of the listed ethnicities. 


Table 3. Percentage of NSC Participant Completions by Ethnicity at each College 


College 

African 

American 

American Indian 
Native Alaskan 

Asian 

Hispanic 

Native Hawaiian 
Pacific Islander 

White 


# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

AACC 

12 

32.4 

2 

66.7 

5 

71.4 

5 

41.7 

- 

- 

21 

50.0 

CCC 

57 

40.1 

0 

0.0 

4 

57.1 

4 

40.0 

- 

- 

34 

49.3 

CLC 

1 

9.1 

1 

50.0 

1 

16.7 

8 

33.3 

- 

- 

16 

22.9 

CPTC 

7 

77.8 

0 

0.0 

15 

93.8 

4 

80.0 

1 

100.0 

50 

87.7 

FSCJ 

9 

15.5 

0 

0.0 

0 

0.0 

3 

20.0 

0 

0.0 

13 

15.5 

MCC 

10 

76.9 

- 

- 

3 

75.0 

- 

- 

1 

100.0 

33 

91.7 

NWACC 

- 

- 

- 

- 

- 

- 

0 

0.0 

- 

- 

4 

20.0 

RSCC 

6 

60.0 

2 

66.7 

1 

100.0 

2 

100.0 

- 

- 

35 

44.3 

SSC 

29 

96.7 

2 

100.0 

27 

90.0 

3 

100.0 

1 

100.0 

42 

89.4 


Table 4 contains data denoting that the percentage of NSC participants completing a program in 
each technical track varied by ethnicity. Asian students completed programs at a higher 
percentage than other ethnicities in Mechatronics, Cyber Technology, and Composites. Only 
37% of white students completed a Mechatronics certification compared to 92% of white 
students completing an Electric Vehicle Technology certificate. Eighty-three African American 
students attained a Composites degree compared to only one earning an Environmental 
Technology certification. Forty-three Asian students completed a Composites program compared 
to 13 in all other technical tracks combined. 

Table 4. Percentage of NSC Participant Completions by Ethnicity in each Technical 


Track 



African 

American 

American 





Native 



Technical Track 

Indian Native 
Alaskan 

Asian 

Hispanic 

Hawaiian 
Pacific Islander 

White 


# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

# 

% 

Mechatronics 

8 

20.0 

3 

50.0 

6 

54.5 

12 

36.4 

0 

0.0 

75 

36.8 

Cyber Technology 

26 

35.1 

1 

100.0 

4 

66.7 

5 

31.3 

- 

- 

28 

36.4 

Composites 

83 

52.5 

2 

33.3 

43 

86.0 

11 

64.7 

- 

- 

108 

75.0 

Electric Vehicle 
Technology 

10 

76.9 

- 

- 

3 

75.0 

- 

- 

1 

100.0 

33 

91.7 

Environmental 

Technology 

1 

5.0 

1 

50.0 

0 

0.0 

1 

16.7 

- 

- 

4 

9.3 


There were three stand-out NSC colleges in terms of program completers: SSC, MCC, and 
CPTC. These colleges had the highest percentage of completers overall as well as when broken 
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down by student demographics. AACC and CCC had lower percentage of completers, but this 
may be a reflection of the larger number of participants enrolled at those colleges. FSCJ and 
NWACC seemed to struggle in terms of program completers, which aligns with the lower 
percentages and enrollment numbers of the Environmental Technology track which was offered 
at both of those colleges. While there were differences in success across colleges and technical 
tracks, as a whole, data suggest NSC was somewhat successful in meeting program completion 
goals for the grant. 

Retention Outcomes 

Another primary focus of the NSC grant was to increase retention rates of participants. This 
section outlines retention and completion of NSC programs within the appropriate timeframe. 
Most NSC programs were one-year, with the exception of the Composites certificates which 
were designed as a six-month program. 

The percentage of TAA-eligible, Priority of Service, and students with developmental needs 
retained in NSC programs can be found in Figure 11. While very few TAA-eligible students 
were enrolled in NSC programs, the vast majority were retained during the grant (89%). Data 
indicate similarly high retention rates for students with Priority of Service and developmental 
needs, suggesting the NSC colleges were successful in meeting project goals focused on 
retention. 



■ Completed Still Enrolled ■ Withdrew 


Figure 11. Retention of TAA eligible, Priority of Service, and Developmental Need 
Participants 

Figure 12 displays the percentages of male and female students retained in NSC programs. 
Although more male students enrolled overall, the percentage of both males and females that 
completed or withdrew from a program is nearly identical, with a slightly higher percentage of 
female students withdrawing from an NSC program. 
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■ Completed Still Enrolled ■ Withdrew 


Figure 12. Retention of Female and Male Participants 

The percentage of students retained in a program differed more by ethnicity than gender. Figure 
13 shows the highest withdrawal rates belonged to African-American students (29%), while the 
lowest withdrawal rates (7%) are of students who identify as Asian. Fourteen percent of Hispanic 
students withdrew from NSC programs and 49% of those included in our sample are still 
enrolled. The completion percentages for Asian and Native Hawaiian/Pacific Islanders were 
above 70%. The completion percentage for students from all other ethnicities ranged from 37% 
for Hispanics to 48% for whites. 
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■ Completed Still Enrolled ■ Withdrew 

Figure 13. Retention of Participants by Ethnicity 

Figure 14 presents the percentage of TAA-eligible, Priority of Service, and developmental needs 
participants who finished their certification within the appropriate timeframe. All TAA-eligible 
students completed their program within the appropriate timeframe, six- months for Composites 
programs and one year for the other technical tracks. Sixty-five percent of veterans finished 
programs within expected timeframes, similar to the 63% of participants with developmental 
needs. 
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Figure 14. Percentage of Participants who Completed within the Appropriate 
Timeframe 

The percentage of male and female students to complete a program within the appropriate 
timeframe is displayed in Figure 15. Although male students represented the majority of 
participants enrolled in NSC programs, 135 female students participated. Female studentsO 
ability to attain a certification within the appropriate timeframe was substantially lower than their 
male peers. Only 18% of female students completed a program within the expected timeframe 
compared to 82% male students. 


Male (n = 716) 


82 % 


Female (n = 135) 


18 % 


Figure 15. Percentage of Female and Male NSC Participants who Completed within the 
Appropriate Timeframe 

The percentage of NSC students to complete a program within the appropriate timeframe is 
displayed by ethnicity in Figure 16. All American Indian/Native Alaskan students attained a 
certification within the appropriate timeframe. The majority of Asian students (88%) completed 
programs within the expected timeframe and 75% of white students, representing the largest 
NSC population, attained a certification within expected timeframes. Only half of Native 
Hawaiian/Pacific Islanders and less than two-thirds of African-American (59%) or Hispanic 
(65%) students completed NSC programs within appropriate timeframes. 
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Figure 16. Percentage of NSC Participants by Ethnicity who Completed within the 
Appropriate Timeframe 


Overall, these data suggest that NSC colleges met retention goals. In addition, the vast majority 
of NSC participants completed their programs within the appropriate timeframe, indicating that 
colleges were successfully helping students get the training needed to enter the workforce or 
continue their education. However, the substantial difference between completion within the 
program timeframe for male and female students should be noted. Whether this is a reflection of 
the program content, inherent characteristics, lack of student support directed to female students, 
or another cause cannot be determined through this analysis. 


Employment Outcomes 

The overarching goal of the NSC is to assist participants in securing new employment by 
equipping them with technical skills and providing job placement support. Achievement of this 
goal has been largely successful, as many completers have found relevant employment. 
According to NSC staff, approximately 70% of completers have found employment. All 
programs have shown some level of success. In particular, those completing the Composites or 
Mechatronics programs have been largely successful in finding employment, citing 70-100% 
placement. 


Across all technical tracks, there have been ample job opportunities, as demand is generally high 
for the skill sets being acquired. Each grant year, employers and program staff expressed 
difficulty filling positions with qualified candidates; however, the NSC certificate programs have 
begun to provide a pipeline of skilled workers to meet the need. In fact, several employers now 
skip the public recruiting process and go directly to their partner NSC college for hiring, which is 
indicative of a high level of satisfaction among employers with the preparedness and quality of 
NSC completers. In Year 4 interviews, several employers stated that NSC completers have 
generally been beneficial hires due to both technical and nsoftd skills. Specifically, they noted 
their ffexceptionalO mechanical aptitude, knowledge of new technology, and troubleshooting 
ability. Many emphasized their positive attitudes, willingness to receive instruction, 
professionalism, and communication skills. These attributes have put many NSC completers 
ahead of other candidates, and have reduced the time needed for on-the-job training. In general, 
employers interviewed agreed that NSC certificate completers tend to be more marketable than 
other job seekers. Some employers, as well as NSC completers surveyed, attributed their 


Hezel Associates, LLC 


31 








employment success with the embedded industry certifications within their technical tracks, 
particularly in the Cyber Technology track. 

Those who completed an NSC certificate program were typically hired for entry-level technician 
positions, with the potential for future advancement; however, the specific job title and 
responsibilities depend on the technical track and employer. Pay rates of these positions vary, but 
most fall in the $10-$30 per hour range. Employers hiring Mechatronics completers reported the 
highest pay rates, ranging from $24-$30 per hour. In Year 4, many employers mentioned that 
they offered paid internships to NSC students. Most treated this as a ntest drived to vet 
candidates and gauge if the student was the right fit. The success rate was high, as most interns 
were offered permanent positions. 

From the studentsOperspectives, both directly and indirectly from program staff comments, they 
have generally been pleased with their experience and ability to secure a job. Program staff often 
receive positive feedback from students who shared that they were able to adjust to their job 
quickly, and that the course content aligned with their actual job duties. 

According to a student questionnaire administered in April and May 2015, of the 38 survey 
respondents who had completed their program, more than three-quarters believed they had more 
employment options than before they enrolled. Specifically, all of those completing the 
Mechatronics certificate indicated more employment options, while responses from those in 
other technical tracks were mixed. More than 80% were employed and doing work related to the 
certificate they received. More than half of those employed did not think they would have been 
hired at their current position without the certificate. This was particularly apparent among those 
who had completed the Cyber Technology certificate. When asked about wage changes after 
receipt of their certificate, responses were equally divided between seeing a wage increase and 
no change. 

Participants who completed their program and were employed were also asked to rate various 
aspects of their program. Overall, responses indicated that completers were somewhat satisfied 
with abilities they gained during their program regarding job searching, soft skills, and technical 
skills. This is consistent with student responses from a similar questionnaire in 2014. Figure 17 
shows the percentage of respondents from the most recent questionnaire who chose strongly 
agree , agree , or somewhat agree that their program prepared them to perform each task. 


Efficiently search for relevant job openings 
Create a professional resume 
Communicate about my abilities in an interview 
Write an effective cover letter 



Figure 17. Completers’ Perception of Job Search Preparation (n = 31) 
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Respondents who were employed rated how well their NSC program prepared them for various 
skill sets on the job, including both soft skills (i.e., communication) and technical skills. Figure 
18 displays the percentage of respondents who chose strongly agree , agree , or somewhat agree 
for each skill. The majority agreed to some extent that they were equipped with each skill, 
particularly the technical skills. However, certain soft skills, like time management, task 
prioritization, and leadership were among the lowest. While soft skills are included in STEM 
Bridge curriculum the technical content is much more prevalent in each track; therefore, it is 
expected that students will retain those subjects more readily. 


Apply technical skills 
Work as a member of a team 
Troubleshoot technical problems 
Operate equipment used in the industry 
Apply quality control knowledge 
Use required computer software 
Apply math skills 
Effectively communicate verbally 
Prioritize tasks 
Apply writing skills 
Manage my time 
Lead groups of colleagues 



Figure 18. Employed NSC Completers’ Employment Preparation (n = 31) 

It was noted by program staff, however, that students were most successful when they showed 
initiative and were proactive in pursuing job search assistance. In particular, those who utilized 
the embedded student support services, including resume writing assistance and mock 
interviewing, were more likely to find employment. Similarly, those with realistic salary and 
position- level expectations were more likely to accept positions they were qualified for. And, 
while the NSC programs are designed as 1-year certificates that will result in employment, some 
students have decided to continue with their education and pursue an associated; or bachelors 
degree. 

While NSC completers have, overall, enjoyed employment success, some employers expressed 
concern that an associated or bachelord degree would be more desirable than the 1-year 
certificate. This is especially true for the Environmental Technology and Electric Vehicle 
Technology tracks, where the higher degree would allow new hires to advance in their careers 
toward management positions. 
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Impact on NSC Colleges 

While much of the exploration of NSC impacts has been in regards to student outcomes, it is also 
important to reflect on how the NSC affected its member colleges. The most obvious and 
substantial change to the colleges was the addition of new programs or the enhancement of 
existing programs. In some cases, the tracks were an expansion of a program or a repackaging of 
courses and content into a more formal curriculum. Others started from ground zeroOand 
established a brand new program, particularly within Mechatronics. These all involved the 
acquisition of new or updated equipment. The short timeframe of these programs were somewhat 
unique within the NSC colleges, with some program staff noting that the condensed format 
worked well. Others, specifically those involved with the Cyber Technology tracks, asserted that 
it is too aggressive of a timeline, with some students not becoming fully prepared for industry 
certification exams. Most staff indicated that the technical tracks will continue to be offered at 
their respective colleges, and curriculum updates and maintenance will be funded through the 
collegesOexisting operating budgets. 

STEM Bridge is a unique feature of the NSC model that did not exist prior to the grant. All NSC 
colleges incorporated this curriculum into their tracks, whether they required it or allowed 
students to test out of portions of it. Some program staff acknowledged the applicability of 
STEM Bridge to non-NSC programs in their colleges. A few colleges applied the curriculum 
outside of NSC and offer STEM Bridge college-wide, while others used it as a Stepping stoneO 
for high school students into college or students transitioning into an associated degree program. 
More than half of the colleges intend to use STEM Bridge after grant end, whether in the NSC 
technical track or in other programs. 

Various aspects of the NSC model were new to the member colleges. For example, the navigator 
position is a new position for most NSC colleges that was made possible by the grant. This type 
of one-on-one support is a new advising format that many could not take advantage of without 
the additional funding provided by the TAACCCT program. Unfortunately, less than half of the 
partner colleges plan to keep the navigator position, as they will not have the financial resources 
to support the role post-grant. The cohort model and block scheduling are other structural 
changes for many colleges. A few colleges indicated that they will continue to implement this 
format for the technical track(s) post-grant, and two colleges plan to expand this format into 
other programs. 

The consortium itself had an important impact on the member institutions, as it has fostered 
partnerships between community colleges that would not have occurred otherwise. The technical 
teams formed across colleges and in some cases, across the country. Project staff have cited these 
relationships as collaborative, helpful, and likely to outlast the grant. In addition, most NSC 
colleges have expanded their relationships with employers by establishing linkages with firms 
they had not worked with before and strengthening collaborations with existing partners. The 
typical role NSC employers play has grown over the course of the grant from an informal 
curriculum advisor to a substantial partner who invests considerable time and resources into the 
program. Many employers have shifted their hiring habits to contact program coordinators or 
navigators directly, instead of publicly posting job openings. Many have developed internship 
programs for NSC students, which gives them real world experience. Therefore, these new and 
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enhanced relationships have not only helped to ensure curriculum and industry are aligned, they 
have given students the opportunity to interact with companies, which has the potential to 
streamline the hiring process after they complete their program. However, with the loss of some 
navigators and program staff post-grant, these relationships could weaken. 

Opportunities for Sustainability and Scale Up 

The final evaluation question was intended to explore the scale up and sustainability of aspects 
of the NSC. Program staff identified several aspects of the NSC model for scaling up, which 
would be applicable outside of the NSC program, both within their respective colleges and at 
other colleges. 

The navigator position, cohort model, and STEM Bridge were mentioned most often as scalable 
both within and outside of NSC colleges; however, each comes with unique challenges. Some 
suggested that the positive impact of the navigator could be realized in many other programs 
within their own colleges. The position could be expanded to other programs in a particular 
department, or more positions could be added to serve other departments and programs. The 
most encumbering challenge to the navigator position is cost, as resources are not often available 
to pay an individual for such in-depth advising and connection to students. The cohort model is 
also beneficial structure for students, instructors, and employers, but implementation could 
require a shift in scheduling and program format that may be conflict with collegesOregistration 
and tracking systems. 

Program staff highlighted STEM Bridge as being easily applicable to other technical programs in 
their own college and others, due not only to its content, but its ease of use on the OLI platform. 
Some have already begun processes to offer STEM Bridge to other programs or their entire 
campus. The scalability of STEM Bridge outside of NSC is already apparent, as it is highly used 
by other institutions, including high schools and middle schools. While this is providing a benefit 
for thousands of students currently, it is unclear who or how the content will be continually 
reviewed and updated without the NSC resources. 

The technical track curricula were designed to be nationally portable; therefore, their scalability 
has been a project focus. Having the entire curriculum, from program guides and teaching 
toolkits to lesson plans and assessments, on OLI and Skills Commons has and will allow access 
to anyone who is interested. That said, there have been other colleges that have reached out 
directly to NSC for advisement and assistance on creating similar programs. This has been 
particularly prevalent with the Mechatronics program, as it is a relatively new field. The Cyber 
Technology curriculum has also been in high demand, and is the basis for a Round 4 TAACCCT 
project in Maryland. Also in regards to curriculum, a few program staff members thought the 
format and model of the development process would be of use to those outside of NSC and hope 
to share that as well. While most technical tracks at the NSC colleges will continue, some plan to 
terminate the programs at grant end, mainly due to low enrollment or a mismatch between the 
curriculum focus and regional employer needs. 

Program staff have continually acknowledged the success of the NSC, in terms of collaboration, 
creation of new relationships among partners, innovative educational solutions, and the ultimate 
employment of students. The group is unclear what the NSC will look like after grant end, but 
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the majority of staff and faculty interviewed would l ik e it to continue in some form. The 
consortium members focused on this in Year 4 and spent considerable time at group meetings 
(e.g., technical team meetings, the NSC Summit) brainstorming about the structure of a 
continued collaboration and how best to secure resources. This resulted in a menu of options to 
consider, including applying for additional grants, becoming an accreditation body, and 
affiliating with a larger organization or foundation focused on similar topics. 

The greatest concern voiced by program staff in relation to sustainability was the future of the 
technical content of the curricula. While individual colleges will likely keep their programs 
current, content recently uploaded to OLI and Skills Commons is static. Without a formal entity 
like NSC to continually review and update content, the OER curricula could quickly become 
stale. This is especially true of STEM content, as technological advances, such as equipment and 
software updates, occur rapidly. Therefore, most interests were centered on maintaining the 
public curricula. However, there is still uncertainty as to how that will be addressed. 

Additional funding is being pursued by a few members to expand certificate programs; however, 
there are no plans to pursue funding for the NSC itself. Regardless of the structure of the group 
in the future, most program staff plan to maintain the connections they have made with the other 
members, particularly within technical teams. 

Track-Specific Sustainability and Scale Up 

Employers interviewed identified future trends in each technical track that may drive how and 
where NSC technical tracks are sustained or scaled. A more detailed labor market analysis is 
available in a series of NSC labor market reports prepared by Hezel Associates. 

Composites. According to employers interviewed, the composites industry is a growing and 
evolving field. Employers partnering with NSC colleges, in both Washington State and the 
Midwest, expect growth and increased needs for certificate-level employees. Not only are 
workers needed in the aviation sector, expansion is anticipated in the marine and automotive 
industries as well. Technological innovations are also expected, including an increase in the use 
of 3D modeling, which will drive the specific skills needed, such as increased math and science 
knowledge. Therefore, the outlook for the NSC Composites certificate program is positive, as it 
is important in both the Pacific Northwest and the Midwest, with potential in other regions as 
marine and automotive needs increase. 

Cyber Technology. While the Cyber Technology industry is growing (Csorny, 2013), employers 
and project staff gave mixed remarks on the future demand of the 1-year certificate. A few 
employers observed that the Cyber Technology certificate is a good way to fgct a foot in the 
door,Obut a candidate would need further education (such as a 2- or 4-year degree) to be hired at 
a mid-level position or to advance in position. Employers noted that experience is often needed 
for hiring, in addition to education. Needs are specific to regions and particular companies, 
however, and most of those completing the NSC certificate have found employment. Therefore, 
if these specific needs are known and integrated into the NSC programs in the future, or into 
newly created NSC-like cyber programs, employersOskill needs would likely be met. 
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Electric Vehicle Technology. While all technical tracksOrelated labor markets are affected by 
external factors, employers and program staff pointed out that Electric Vehicle Technology is 
particularly influenced by politics and the overall economy. The demand for electric vehicle 
workers fluctuates in step with fuel prices, as consumer demand for these vehicles increases with 
increased fuel costs, which creates a need for more technicians. The opposite is also true, where 
low fuel prices translate into lower electric vehicle demand. With no strong state or federal 
policies regarding this technology, this will likely continue to be the case. Also noted by 
employers and project staff, it is typically more advantageous for workers to have a 2- or 4-year 
degree, as opposed to the 1-year certificate. Therefore, the future of this industry is somewhat 
unclear, but has been fruitful for students so far. 

Environmental Technology. Project staff and employers reported that the environmental industry 
is strong and appears to be growing, particularly in the South and Midwest. While some NSC 
completers have found employment, the 1-year certificate may not be the best fit, as employers 
typically prefer a 2- or 4-year degree. Also noted was the regional- specific needs of certain 
tracks within the environmental pathway; for example, the hazardous materials track, as opposed 
to the water quality track, was found to be a better fit for the Midwest region after the water 
quality track was in place at an NSC college in the region. While the industry is relatively steady 
and potentially growing, new 1-year certificate programs in this field will need to be carefully 
considered, with substantial input from regional industry, to be successful. 

Mechatronics. The need for workers with mechatronics training is strong and expanding. 
Employers are actively seeking workers with mechatronics training and find that the 1-year 
certificate provides the skills needed, including precision machining, troubleshooting ability, 
controller knowledge, and hydraulics training. This field has considerable promise for the 
foreseeable future and the 1-year certificate appears to be an appropriate level of training. 
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CONCLUSIONS 

The NSC successfully met expectations for all strategies and activities outlined in their work 
plan, as well as completed anticipated deliverables. These included the formation of a structure 
and communication scheme for the consortium, establishment of new partnerships, design of five 
technical tracks and STEM Bridge curriculum, implementation of those tracks and enrollment of 
students, collection and analysis of data pertaining to the grant work, and the development of 
sustainability and scale up plans. While not all items were completed within the initially 
proposed timeframe, all were completed during the grant period and delays were due to typical 
lag times in staff hiring as well as USDOL approval delays. 

In terms of specific findings from evaluation of the 4-year grant project, Hezel Associates has 
concluded the following: 

• The grant was well managed. In terms of establishment of consortium-wide policies 
and regular communication, the NSC grant management team was well organized. 
Program staff members at each consortium college were pleased with the ease and 
frequency of conference calls, technical team meetings, and annual NSC Summits. The 
team was also adept at adhering to USDOL policies and meeting procedural, fiscal, and 
reporting requirements. 

• Development of technical track curricula was a truly collaborative effort, with a 
focus on easy expansion outside of NSC. The development of curricula for the five 
pathways constituted the bulk of the NSC work, involving dozens of partners. Partners 
included NSC faculty and staff, NSC management and the curriculum development team, 
SMEs (education consultants and industry experts), and local and regional employers. 

The curriculum development team was responsible for ensuring content was presented 
consistently, so they provided templates to developers. Aspects of each program curricula 
regarding the implementation were also created, including teaching toolkits, sticking 
points, and program guides. These were intended to provide an institution outside of NSC 
with tools to deliver the curriculum in a similar manner, not only including technical 
content, but the educational model as well. Finally, the curricula were always intended to 
be OER, and their availability on the OLI platform and Skills Commons has ensured they 
are in the public arena. NSC staff has also spent substantial time promoting its curricula, 
directing institutions to OLI at conferences and other educational meetings. Therefore, 
the curriculum content is likely of high quality, since educational professionals, SMEs, 
and employers have shared positive feedback. Its adoption has been made as easy as 
possible, with the creation of implementation guides. And, this high quality content is 
now free and available to other institutions interested in adopting one of these programs. 

• STEM Bridge has garnered considerable interest outside of NSC. The integration of 
the STEM Bridge modules into the NSC technical tracks has been identified by program 
staff as a useful refresher and an effective way to address studentsOneeded skills, 
particularly math and reading. The transition of this curriculum into the public domain on 
the OLI platform and Skills Commons has led to use from a variety of institutions, 
including non-NSC colleges, high schools, and middle schools. Use has increased 
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steadily since it was made public, and is expected to continue. This demonstrates the 
scalability of the modules and their applicability to other programs and formats. 

• NSC institutions have enhanced their relationships with local and regional 
employers over the course of the grant. NSC member colleges placed substantial 
emphasis on developing employer relations. Some of these existed before the grant, but 
many were created under the grant through targeted outreach by program coordinators, 
faculty, and navigators. Regardless of whether a relationship was new or not, most were 
strengthened over the 4 years. Employers played a key role in curriculum development 
and implementation, serving on NSC-specific advisory boards at each college, acting as 
guest speakers, advising on appropriate equipment purchases, and providing professional 
development for faculty. These collaborations were mutually beneficial, as the colleges 
and technical teams were able to develop industry- re levant content, while the employers 
involved were able to ensure an appropriately skilled pipeline of new workers. 

• The NSC model fostered student success. The elements of the NSC model, including 
block scheduling, cohort enrollment, a compressed timeline, employer linkages, hybrid 
delivery, STEM Bridge, and the tailored student support (the navigator), likely played a 
key role in student success. NSC student term-to-term retention was 11% higher than 
typical community college students, and the completion rate was almost 70% higher. It is 
difficult to identify any one element that was most important in fostering student 
achievement; however, program staff attributed these numbers mainly to the tailored, 
one-on-one support from the navigators, as well as the peer support present in cohort 
enrollment. Students also voiced gratification with the employer involvement in the 
program development and delivery. A more detailed examination of the model is 
available in the NSC Model Report. 

• NSC was on target for most of its participant goals. NSC enrollment exceeded its goal 
of 1,279 participants, with 1,372 students beginning a technical track. Of these, 656 
students completed their program, which is 64% of the overall project target of 1,023. 
NSC students exceeded the target retention rate of 80%, at 82%, which is also higher than 
the national community college average of 75%. Enrollments and completions at the end 
of the grant period were likely higher than reported here; however, these data were not 
available at the time of reporting. 

• Female students took longer to complete their technical tracks than males. 

According to the data used for the quantitative analysis, 18% of female NSC students 
completed their program within the expected timeframe, in contrast to 82% of male NSC 
students. It is not clear why this difference occurred, but may be related to the program 
content, inherent student characteristics, or lack of student support directed to female 
students. In addition, a more complete analysis of all NSC student data may show 
different results. 

• NSC completers were generally successful in finding employment. According to NSC 
staff, approximately 70% of completers have found employment, and those in the 
Composites and Mechatronics field seemingly have the highest success rates. As these 
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technical tracks were designed to address particular needs in regional labor markets, it is 
not surprising that there have been positive employment outcomes for students. However, 
the response of employers has been overwhelmingly positive, with many expressing 
excellent feedback on NSC hires, in terms of technical skills and professionalism. 
Because many employers are now contacting the technical track staff and faculty directly 
for hiring, instead of publicly posting job openings, these programs have created a 
pipeline of talent that employers appreciate and actively look for. 

• The NSC’s future is unclear. Sustainability was a major focus of the final grant year. 
Through collaborative work of NSC college leaders, NSC staff and faculty, and external 
groups, the team identified their priorities and numerous ways to continue their work. 
However, the consortium as a whole does not have a formal plan to continue as it is 
currently. Without USDOL funding, it is expected that some colleges or individuals will 
discontinue participation, but many of the established relationships and idea-sharing will 
continue informally. These informal partnerships may allow technical teams to keep the 
curricula up to date within the respective NSC colleges; however, the content housed on 
OLI and Skills Commons may become stale without a plan to continually review and 
update it. Some NSC members are teaming up to pursue additional funding, which is 
intended for further curriculum development (e.g., an associated, degree). 
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RECOMMENDATIONS 

While the funding for the NSC project has ended and the formal consortium structure will no 
longer exist, Hezel Associates offers the following recommendations for individuals interested in 
maintaining aspects of the collaboration, or for further research. 

• Focus on what model aspects work best in each institution. Each NSC college has 
unique characteristics, considering they are spread across the country in different states. 
Local and regional labor markets, student populations, institutional culture, state 
regulations, and resources, among other things, differ. While the NSC model elements 
were originally intended to be implemented cohesively across NSC, some colleges made 
slight changes to accommodate specific needs. For example, some were not able to 
implement block scheduling due to a conflict with their registration and scheduling 
system. Others limited the amount of online coursework due to a need for hands-on work 
necessitated by the nature of the content or by industry certification requirements. These 
sorts of adjustments should be considered to allow the program to fit into the institutional 
culture and structure, as well as to ensure it works for an institution® specific student 
population. The intent of the model is not to be rigid, but to use elements as they fit to 
garner the most effective program for student success. Moving forward, the NSC colleges 
should continue to revisit their program structure and elements to ensure the best 
situation for their students, as well as to serve as a model for other institutions interested 
in adapting the format. 

• Continue to collect data and analyze the effectiveness of the model’s components. 

Related to the first recommendation, institutions can only make informed decisions on 
what elements work best with reliable evidence. Hezel Associates has provided a 
supplemental report, entitled NSC Model Report, which delves deeper into the elements 
and their effectiveness at NSC colleges and beyond. This is a first look at how pieces of 
the model functioned within the NSC, and how the model relates to the current state of 
community college research. Therefore, this preliminary research can be expanded 
substantially, and NSC members are encouraged to keep abreast of the literature 
regarding the elements, as well as to determine key indicators that measure success in 
their own institutions. This could entail individual research at the instructor level, perhaps 
with student support; pursuit of funding to perform more structured research studies; or 
collaboration with other institutions (NSC or non-NSC) to share data and expertise via 
conferences or coordinated research. Because NSC colleges are well positioned to be 
leaders in these innovative model elements, their knowledge and contribution to the 
current and future understanding of their effectiveness can help to shape opportunities to 
improve community college educational models. 

• Identify leaders who will maintain NSC curriculum in the absence of a solid 
partnership. While the grant requirement to transfer curricula to OER has been fulfilled, 
the NSC has a desire to keep its work current in the future, as STEM programs in 
particular can become outdated quickly. Without a formal consortium structure moving 
forward, the maintenance and updating of the curricula on OLI and Skills Commons is in 
question. It is likely that individual NSC institutions will maintain and update their 
curricula internally and will informally consult with fellow technical team members at 
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other NSC colleges when doing so. It would be to their advantage to keep materials 
current in the public arena, as their colleges and names are identified as the authors, 
thereby conveying a sense of fbwnershipOand giving interested parties a person or 
institution to contact with questions. Technical team leads may be a source for editing 
curricula on OER, as they are most likely to be aware of changes that occur. While a 
formal agreement may be difficult without the NSC as a facilitator, agreements among 
the technical team colleges themselves may be possible. This may be a short-term 
solution, if in fact the NSC identifies a way to continue in some form in the future. 

• Pursue additional grant funding. The most evident obstacle to the continuation of the 
NSC work is the lack funding and other resources. Pursuit of additional grant funding 
could be worthwhile; however, care should be taken to ensure the funding supports the 
aspects of the work that are most valuable. For example, requesting funds to create an 
associated degree that is stackable with a certificate in an NSC technical track and is 
desired by regional employers, creating a direct pathway for students to continue their 
education, aligns well with the intention of the NSC work. 

Potential funders include the National Science Foundation, particularly the Advanced 
Technology Education program; the Bill & Melinda Gates Foundation; as well as several 
STEM-specific opportunities listed on the U.S. Department of Education website 
(http://www2.ed.gov/about/offices/list/ovae/pi/cclo/stem.html) . The American 
Association of Community Colleges (2015) also recently launched a project to help 
community colleges build career pathways, in the form of informational institutes for a 
select group of colleges, expanding outward in later years. This opportunity is worth 
monitoring, may benefit from NSC expertise, and could generate funds and resources in 
the future. 

Any request for funds that build on NSC work should leverage evidence generated during 
the grant period. These data could be internal, such as student retention and completion 
rates, or from the conducted research and evaluation. Those submitting any grant 
applications are encouraged to use data that are available, such as evaluation reports, 
which will be accessible on Skills Commons. 
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APPENDIX A: YEAR 4 INSTRUMENTS 


Document Review Framework 
NSC TAACCCT Round 1 

Qualitative research to assess fidelity with which program activities were 
implemented and in compliance with the timeline. 

Data collection and analysis will be conducted annually. 

Documents will be collected through the NSCG; Program Director and other 
NSC staff. Documentation will be provided to Hezel Associates via email, 
through NSC 6; Sharefile system, or secure file transfer protocol (SFTP) using 
Hezel AssociatesOinternal server dependent upon the sensitivity of the 
documentation. 

The activities in the work plan will guide the identification of documentation 
to use as evidence. 

Once documents have been collected and sorted, content in each document 
will be examined and entered in the following matrix aligned with the 
appropriate milestones. Hezel Associates will list each document, what AACC 
has done to justify fulfilling that milestone, and the time period it was fulfilled 
under Evidence. Status for meeting the listed milestones will be marked Met, 
Not Met, or In Progress, with description of timeline fidelity. 

Instructions Please provide documentation supporting milestones, activities, and 

deliverables listed in the following matrix, including any evidence of program 
implementation and compliance with project implementation timeline. 
Documents can be submitted as attachments via email or using Hezel 
AssociatesOinternal server via SFTP if documents contain sensitive 
information. All document names and a description of each document should 
be included in the table on the first page. Hezel Associates will fill in nStatus,0 
and ft vide need boxes during analysis. 

Definitions Activity: Activities as listed in the work plan included in the technical 
proposal 

Year(s): Grant year of which the activity is to be completed 
Milestone: Project milestones as defined in the work plan 
Time Period: Specific date (or date range) when milestone is to be met 
Status: Status for meeting milestones: Met, Not Met, In Progress, with 
description of timeline fidelity 

Evidence: Document providing evidence of milestone, explanation for how 
the milestone was fulfilled, and in what time period. Description of each 
document is provided in the following table. 


Format 

Timeline 

Process 
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Strategy 1: Design new curricula within “built-for-completion” pathways that integrate whole program design with cohort enrollment, block scheduling, 
compressed classroom time with hybrid delivery, and embedded student support and employer linkages. 

Activity 

Year(s) 

Time 

Period 

Deliverables 

Milestone 

Status 

Evidence 

1 . Organize project coordination, 
policies, and procedures for 
start up and implementation of 
the project. 

1 

3/31/12 

NSC policies; fiscal 
procedures; project 
manager, other grant staff 
and technical consultants; 
meeting of steering 
committee, work teams, 
and cross-cutting teams 
established. 

Finalize staff needs 



Procure technical expertise 



Establish fiscal procedures 



Determine reporting and 
data collection procedures 



Evaluation designed and 
outside evaluators secured 



2. Establish governance and 
partnerships to integrate deep 
involvement of higher 
education, industry, workforce 
development, and labor. 

1.2,3 

3/31/12 

Governance structure with 
top leaders from higher 
education, business, 
industry, workforce 
development. 

Partnerships developed 
among key stakeholders 



Governance structure 
finalized 



3. Design unified curriculum for 
each STEM certificate pathway 
and deliver professional 
development to ensure quality 
of instruction, competency 
based learning, student support 
services. 

1,2 

12/31/12 

Five STEM Certificates 
(composites technology; 
cyber technology; electric 
vehicle technology; 
environmental technology; 
mechatronics) all industry 
validated & nationally 
portable. New set of 
program delivery methods, 
hybrid technologies & 
student support services 

Continued statewide and 
national reconnaissance to 
inventory/acquire the best 
curriculum 



Procure content-specific 
technical assistance to 
develop the curriculum 
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Strategy 1: Design new curricula within “built-for-completion” pathways that integrate whole program design with cohort enrollment, block scheduling, 
compressed classroom time with hybrid delivery, and embedded student support and employer linkages. 


Activity 

Year(s) 

Time 

Period 

Deliverables 

Milestone 

Status 

Evidence 




integrated into each 
pathway. 

Guide for faculty and staff 
on STEM certificates. 
Professional development 
for faculty and staff. 

Purchase equipment & 
materials to enhance 
curriculum content and 
delivery 



4. Outreach to students to ensure 
strong enrollment and 
partnerships with employers to 
secure employment 
opportunities. 

1,2,3 

9/30/14 

Outreach plan and 
materials; employer 
engagement; WIB 
engagement; cohort of 
students. 

Develop outreach plan 



Engage employers to 
secure involvement in 
curriculum and hiring 



5. Implement 5 new one-year 

STEM certificates. 

2,3 

9/30/14 

Students enroll, persist at 
key junctures, complete 
credentials, and gain 
employment in field. 

Enroll cohorts 



Students persist and 
complete credentials 



Students get hired 



6. Identify and resolve issues that 
hinder success and institute 
new policies to sustain and 
build to scale the new program 
designs and pathways across 
states. 

1,2,3 

9/30/14 

Policies to support ongoing 
success and sustainability 
of consortium’s efforts. 

Identification of key policy 
issues to help overcome 
barriers to institutional, 
student, and employer 
success 



Identification of new policy 
opportunities and resources 
for sustainability 
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Strategy 1: Design new curricula within “built-for-completion” pathways that integrate whole program design with cohort enrollment, block scheduling, 
compressed classroom time with hybrid delivery, and embedded student support and employer linkages. 


Activity 

Year(s) 

Time 

Period 

Deliverables 

Milestone 

Status 

Evidence 





Data collected and shared 
annually on progress 
measures and DOL 
outcomes 



Engage in data analysis, 
continuous learning, and plan 
for sustainability. 

1,2,3 

9/30/14 

Report to USDOLon 
measures and outcomes; 
distribution of evaluation 

Host one face-to-face 
national meeting each year 



findings to stakeholders; 
learning and sustainability 
plans distributed 

Frequent virtual meetings 







Attend USDOL events 
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Strategy 2: Create an accelerated STEM Bridge that bypasses traditional developmental education by contextualizing remediation within the programs, 
quickly building the math, reading/writing, computer skills, and critical thinking skills of underprepared students. 

Activity 

Year(s) 

Time 

Period 

Deliverables 

Milestone 

Status 

Evidence 

1. Design & implement STEM 

Bridge that serves each STEM 
pathway. 

1.2,3 

9/30/14 

STEM Bridge that 
integrates basic skills, 
workforce skills, computer 
skills, and job readiness 
training, contextualized 
within each pathway. 

Data collected and shared 
on progress measures and 
outcomes related to STEM 
Bridge 



2. Evaluate STEM Bridge, engage 
in data analysis, continuous 
learning, and plan for 
sustainability and national 
portability. 

1.2,3 

9/30/14 

Consortium-approved and 
externally-validated 
remediation curricula and 
materials aligned with 
learning objectives of the 
STEM programs. 

Evaluation report on STEM 
Bridge. National 
dissemination. 

Data collected and shared 
on progress measures and 
outcomes related to STEM 
Bridge 
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Student Questionnaire 

NSC TAACCCT Grant Evaluation-Year 4 

Page 1 

In 2011, a group of 10 colleges across the country received a US Department of Labor grant to 
create new technical certificate programs that incorporate educational strategies that support 
student success. These programs were launched in 2012-2013. As a participant in one of these 
certificate programs, your feedback is important to help us understand the effectiveness and 
value of these strategies and programs. 

Hezel Associates is looking for feedback on your program. 

This questionnaire will take approximately 10-15 minutes. Be assured that your individual 
responses are confidential and will be reported only as part of group feedback. 

Page 2 

1. Are you 18 years of age or older? 

o Yes 

o No [Go to Termination Page] 

[Required question] 

Page 3 

2. Which college have you attended since fall 2012? Choose one. If you have enrolled in 
more than one of these colleges, please choose the one where you enrolled most recently. 

[Required question] 

o Anne Arundel Community College 
o Clover Park Technical College 
o College of Lake County 
o Cuyahoga Community College 
o Florida State College at Jacksonville 
o Ivy Tech Community College 
o Macomb Community College 
o Northwest Arkansas Community College 
o Roane State Community College 
o South Seattle Community College 
o None of these [ Go to Termination Page] 
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Page 4 

3. Which certificate program did you enroll in since fall 2012? Choose one. If you have 
enrolled in more than one program, please choose the one that you enrolled in most recently. 
[ Only response options for college selected in Q2 will be visible] 

[Required question] 

o Anne Arundel Community College 

a. Cyber Technology 

b. Mechatronics 

c. Other. 

o Clover Park Technical College 

a. Composites Technology 

b. Other. 

o College of Lake County 

a. Environmental Technology 

b. Mechatronics 

c. Other 

o Cuyahoga Community College 

a. Composites Technology 

b. Cyber Technology 

c. Mechatronics 

d. Other. 

o Florida State College at Jacksonville 

a. Cyber Technology 

b. Environmental Technology 

c. Mechatronics 

d. Other. 

o Ivy Tech Community College 

a. Cyber Technology 

b. Electric Vehicle Technology 

c. Environmental Technology 

d. Mechatronics 

e. Other 

o Macomb Community College 

a. Electric Vehicle Technology 

b. Other. 

o Northwest Arkansas Community College 

a. Cyber Technology 

b. Environmental Technology 

c. Other. 

o Roane State Community College 

a. Mechatronics 

b. Other. 

o South Seattle Community College 

a. Composites Technology 

b. Other 
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Page 5 


4. Which best describes your work experience before you began the certificate program? 

o Did not have any prior work experience 

o Experience in my program field or a field similar to my program 
o Experience in an unrelated field 

5. Before enrolling in your program, what was the highest level of education you 
completed? 

o Completed some high school 
o High school diploma or equivalent 
o Some college 
o Earned a certificate 
o Associated (2-year) degree 
o Bachelor® (4-year) degree 
o Master® degree 
o Doctoral degree 
o Other 

6. Are you still enrolled in the certificate program? 

o Yes [Go to Q18] 
o No 
o Unsure 
[Required question] 

Page 6 

7. Did you... 

o Complete the program (earn a certificate)? [Go to Q9] 
o Withdraw from the program without completing? 
o Transfer to another program before completing? [Go to 0181 

o Other [Go to Q9] 

[Required question] 

Page 7 

8. Why did you withdraw from the program? Mark all that apply. 
o Certificate program was different than expected 

o Completed what I intended to 
o Conflict with work schedule 
o Decided certificate program was not what I wanted 
o Difficulty with coursework and/or certificate program requirements 
o Family obligations 
o Financial difficulties 
o Found a job 
o Medical issues 
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o Transferred to another college 
o Transferred to another program at the college 
o Prefer not to answer 

o Other 

[Go to Q18J 

Page 8 

9. How would you describe the changes, if any, to your employment options (e.g., number 
of jobs you qualify for) since you completed your program? 

o My employment options stayed the same 
o I had more options for employment than before 
o I had less options for employment than before 
o Unsure 

Page 9 

10. Choose which best describes your employment status since completing your certificate 
program. 

o I am working at the same company I was at before I started the program, 
o I am working at a different company than I was working at before I started the program. 
[Go to Q12J 

o lam not employed. [Go to Q18J 
[Required question] 

Page 10 

11. After completing the certificate program, which best describes your status with your 
company? 

o I have the same job I had before I started the program, 
o I was promoted. 

o I was laterally transferred (i.e., a move to an equivalent role, usually with a similar salary 
range and a job title at the same level), 
o I was demoted, 
o Unsure 

Page 11 

12. Is your current employment related to your certificate program? 

o Yes 
o No 
o Unsure 

[If Q10 = “lam working at the same company..., ” go to Q15] 
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Page 12 


13. Do you think you would have been hired for your current job without the training you 
received in your program? 

o Yes 
o No 
o Unsure 

14. Did you have to move your residence to take this job? 

o Yes 
o No 

Page 13 

15. How would you describe the changes, if any, to your wages since you completed your 
program? 

o My wages increased 
o My wages stayed about the same 
o My wages decreased 
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Page 14 


16. Please indicate how much you agree or disagree with the following: 

My certificate program prepared me to... 



Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neither 
Agree Nor 
Disagree 

Somewhat 

Agree 

Agree 

Strongly 

Agree 

Not 

Applicable 

a. Communicate about my 
abilities in an interview 

2 

2 

2 

2 

2 

2 

2 

2 

b. Create a professional 
resume 

2 

2 

2 

2 

2 

2 

2 

2 

c. Efficiently search for 
relevant job openings 

2 

2 

2 

2 

2 

2 

2 

2 

d. Write an effective cover 
letter 

2 

2 

2 

2 

2 

2 

2 

2 
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17. Thinking about your current employment, please rate how much you agree or disagree with the following statements: 


My certificate program prepared me with the ability to do the following in a work setting: 



Strongly 

Disagree 

Disagree 

Somewhat 

Disagree 

Neither 
Agree Nor 
Disagree 

Somewhat 

Agree 

Agree 

Strongly 

Agree 

Not 

Applicable 

a. Apply math skills 

2 

2 

2 

2 

2 

2 

2 

2 

b. Apply quality control 
knowledge 

2 

2 

2 

2 

2 

2 

2 

2 

c. Apply technical skills 

2 

2 

2 

2 

2 

2 

2 

2 

d. Apply writing skills 

2 

2 

2 

2 

2 

2 

2 

2 

e. Effectively communicate 
verbally 

2 

2 

2 

2 

2 

2 

2 

2 

f. Lead groups of colleagues 

2 

1 

1 

1 

1 

2 

2 

2 

g. Manage my time 

2 

1 

1 

1 

1 

2 

2 

2 

h. Operate equipment used in 
the industry 

2 

2 

2 

2 

2 

2 

2 

2 

i. Prioritize tasks 

2 

2 

2 

2 

2 

2 

2 

2 

j. Troubleshoot technical 
problems 

2 

2 

2 

2 

2 

2 

2 

2 

k. Use required computer 
software 

2 

2 

2 

2 

2 

2 

2 

2 

1. Work as a member of a 

team 

2 

2 

2 

2 

2 

2 

2 

2 
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Page 15 


18. Have you used or participated in any of the following during your certificate program? 


How satisfied are/were you with the usefulness of each? 



Did you use or , 

participate in... 

If yes, how satisfied were you with its usefulness? 


Yes 

No 

Unsure 

Prefer 

not to 

answer 

Very Dis- 
satisfied 

Dis- 

satisfied 

Somewhat 

Dis- 

satisfied 

Neutral 

Somewhat 

Satisfied 

Satisfied 

Very 

Satisfied 

a. College-organized 
visits to local 
employer facilities 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

b. Counseling 
services 












c. Grant navigator 
(or program 
coordinator) 
assistance 

2 

2 

2 

2 

2 

2 

2 

2 

* 

2 

2 

d. Interns hip(s) 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

e. Job fairs 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

f. Job shadowing 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

g. Mock interviewing 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

h. Online coursework 
(other than STEM 
Bridge) 

2 

2 

2 

2 

i 

* 

* 

i 

2 

* 

* 

i. Speed 
interviewing 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

j. STEM Bridge 
modules 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

k. Tutoring 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 
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Page 16 


19. How satisfied were you with the following components of your certificate program format? 



Very Dis- 
satisfied 

Dis- 

satisfied 

Somewhat 

Dis- 

satisfied 

Neutral 

Somewhat 

Satisfied 

Satisfied 

Very 

Satisfied 

Not 

Applicable 

a. 1 -credit course modules 

2 

2 

2 

2 

2 

2 

2 

2 

b. Block scheduling 

2 

2 

2 

2 

2 

2 

2 

2 

c. Completing all courses with 
the same group of students 
(a cohort) 

2 

2 

i 

2 

i 

2 

i 

i 

d. Compressed schedule (2- or 
3-semester program) 

2 

2 

2 

2 

2 

2 

2 

2 

e. Integration of hands-on 
training in the courses 

2 

2 

2 

2 

2 

2 

2 

2 

f. Mix of online and in-person 
instruction 

2 

2 

2 

2 

2 

2 

2 

2 
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Page 17 


20. During your time in the program, did you successfully complete any industry 
certifications (e.g., OSHA, Cisco)? 

o Yes 
o No 
o Unsure 

21. What is your age? Numeric answers only. 


22. Do any of the following apply to you? 



Yes 

No 

Unsure 

Prefer not 

to answer 

Pell Grant recipient 

2 

2 

2 

2 

Student with a disability 

2 

2 

2 

2 

Trade Adjustment Assistance (TAA)-eligible 

2 

2 

2 

2 

Veteran or Spouse eligible for Priority of Service 

2 

2 

2 

2 


23. What is your gender? 

o Male 
o Female 

o Prefer not to answer 

24. Which of the following best describes you? 

o American Indian/ Alaska Native 
o Asian 

o Black/ African American 
o Hispanic/Latino 

o Native Hawaiian/Other Pacific Islander 
o White 

o Prefer not to answer 
o Other 


Completion Page 

Thank you for completing the questionnaire! 

Termination Page 

Unfortunately, your responses do not meet the criteria for this questionnaire. Thank you for 
participating! 
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Consortium Member Interview Protocol 
NSC TAACCCT Grant Evaluation-Year 4 


Format 


Targets 

Evaluation Questions 


Timeline 


Qualitative research to collect opinions, and will span a broad range of 
issues regarding: 

• Participant employment outcomes 

• Program components 

Semi- structured interview protocol outlines pre-determined questions, 
and allows the interview to probe and pursue unplanned tangents as 
conversations warrant. 

Respondents will be recruited via email. 

Respondents will be faculty and staff members involved in program 
development and implementation. 

Interview questions will address the following evaluation questions: 

3. What impact did the NSC programs have on participantsO 
employment outcomes? 

4. What impact did the NSC program components have on consortium 
colleges in terms of program and course offerings, student 
enrollment, and college processes? 

5. What scale up opportunities exist for consortium colleges? 

Interviews will take approximately 30-45 minutes and will be conducted 
by telephone in February 2015. 
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Initial Recruiting Email 

The National STEM Consortium (NSC) has partnered with Hezel Associates, a research firm in 
Syracuse, NY, to conduct the independent evaluation of Year 4 of the USDOL TAACCCT 
Round 1 grant. 

As a part of our responsibilities, we will be conducting phone interviews with representatives 
from the NSC programs. You have been selected as a potential participant due to your 
involvement in NSC. The purpose of our evaluation is to provide feedback to the NSC Project 
Director and to help improve grant funded activities. 

Telephone interviews will require 30-45 minutes. We are scheduling interviews between [specify 
date range]. Please respond to this email with times and dates if you are available to participate 
in an interview during this timeframe. We will send you a return email confirming your 
scheduled interview. 

This evaluation is being coordinated with Susan Gallagher, NSC Project Director, at Anne 
Arundel Community College. If you have any questions about the evaluation or interviews, she 
can be reached by email at sgallagher5@aacc.edu. You are also welcome to contact me if you 
need more specific information regarding details of the evaluation study. 

Thank you for your support. 

Sincerely, 

[SIGNATURE OF SENDER] 

Pre-Interview Confirmation (via email), with Informed Consent Attachment 

Thank you for agreeing to participate in the NSC grant evaluation process. 

As part of the NSC project evaluation, Hezel Associates will be interviewing program staff to 
explore the grant 6, program components and outcomes. 

Your interview has been scheduled for: 

[INSERT DATE / TIME] 

We will call you at [INSERT PHONE #]. We expect the interview will last 30 to 45 minutes. 

Your individual responses will be kept confidential and aggregated for the report. No personally 
identifying information will be reported, and we will make every effort to protect your identity 
when we present our findings. Please review the Informed Consent document attached to this 
email prior to the interview. 

If you have any questions about the evaluation or your participation feel free to contact me, 

Susan Gallagher, or you may email Solutions IRB (our external review board charged with 
ensuring we treat evaluation study participants ethically) at participants@solutionsirb.com . 

Thank you for your participation, 

[SIGNATURE OF SENDER] 
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Interview Instructions 

ITEMS IN ITALICS SHOULD NOT BE READ TO INTER VIEWEE 


Phone Interview Introduction 

Hello, this is from Hezel Associates. iQn calling about the interview we 

have scheduled to discuss your involvement with the National STEM Consortium project at 
[college]. 

Is now still a convenient time to talk? 

As a reminder, your responses will be kept confidential and aggregated for the report. No 
personally identifying information will be reported, and we make every effort to protect your 
identity when we present our findings. You can stop the interview at any time and skip any 
questions you are not comfortable answering. You can also choose to withdraw your responses. 

Have you read the informed consent document that was emailed to you? 

IF NOT, GO OVER THE MAJOR SECTIONS WITH THEM, ESPECIALLY BENEFITS AND 
RISKS. 

Do you have any questions about the consent form or the study? 

Do you agree to participate in the interview? 

I would like to record our interview to support my note-taking, and the recording will not be used 
for any other purpose. May I have your permission to record our conversation? 

IF PARTICIPANT DECLINES RECORDING, RESEARCHER WILL TAKE NOTES. 
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Questions 


1. Please clarify which technical track(s) you are involved in, if any. 

2. To start, please describe your role in the NSC grant activities. 

(For interviewer: NSC=National STEM Consortium, funded by USDOL TAACCCT) 

3. Thinking specifically about programs funded by NSC, what are the changes that have 
resulted from the NSC funding at your college? 4 

(what the program looks like) 

Grant leadership only: What are the changes that youd'c seen at other colleges in the 
consortium? 2 

In what ways could these changes be applicable outside of the [ technical track(s)] at your 
college? How about outside of the [ college ]? 5 

4. What are the employment opportunities that are available in your region for those completing 
the [ technical track ] at your college? 3 

5. How successful are [ technical track(s)] completers at finding relevant employment? 3 

a. Why do you say that? 3 

6. What kinds of feedback do you receive from those who completed the [ technical track(s)], if 
any? 3 

7. What is your overall opinion of [ technical track ] at [college]? 3 4 5 

What kind of impact do you think it will have on the labor market in your region? 3 - 5 
Thank you, thatd it for my questions, 

8. Is there anything else youQd like to say about the [ technical track] or the NSC project in 
general? 
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Employer/Industry Stakeholder Interview Protocol 
NSC TAACCCT Grant Evaluation-Year 4 


Format 


Targets 

Evaluation Questions 


Timeline 


Qualitative research to collect opinions, and will span a broad range of 
issues regarding: 

• Employer role in grant activities 

• Skills of NSC completers 

• Future relationship 

Semi- structured interview protocol outlines pre-determined questions, 
and allows the interviewer to probe and pursue unplanned tangents as 
conversations warrant. 

Respondents will be recruited via email. 

Respondents will be employer and industry stakeholders involved in 
program development. 

Interview questions will address the following evaluation questions: 

6. What impact did the NSC programs have on participantsO 
employment outcomes? 

7. What impact did the NSC program components have on consortium 
colleges in terms of program and course offerings, student 
enrollment, and college processes? 

8. What scale up opportunities exist for consortium colleges? 

Interviews will take approximately 20-30 minutes and will be conducted 
by telephone in February 2015. 
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Initial Recruiting Email 

The National STEM Consortium (NSC) has partnered with Hezel Associates, a research firm in 
Syracuse, NY, to conduct the independent evaluation of Year 4 of the USDOL TAACCCT 
Round 1 grant. 

As a part of the evaluation, researchers will be conducting phone interviews with employers and 
other stakeholders from the NSC programs to better understand how grant activities align with 
industry needs. You have been selected as a potential participant due to your involvement in the 
grant activities. The purpose of this evaluation is to provide feedback to the NSC Project 
Manager and to help improve grant-funded activities. 

Telephone interviews will require 20-30 minutes. We are scheduling interviews between [specify 
date range]. Please respond to this email with times and dates if you are available to participate 
in an interview during this timeframe. We will send you a return email confirming your 
scheduled interview. 

This evaluation is being coordinated with Susan Gallagher, NSC Project Director, at Anne 
Arundel Community College. If you have any questions about the evaluation or interviews, she 
can be reached by email at sgallagher5@aacc.edu. You are also welcome to contact me if you 
need more specific information regarding details of the evaluation study. 

Thank you in advance for your support as we move forward with this important study. 

Sincerely, 

[SIGNATURE OF SENDER] 

Pre-Interview Confirmation (via email), with Informed Consent Attachment 

Thank you for agreeing to participate in the NSC grant evaluation process. 

Your interview has been scheduled for: [INSERT DATE / TIME] 

We will call you at [INSERT PHONE #]. We expect the interview will last 20 to 30 minutes. 

As part of the NSC project evaluation, Hezel Associates will be interviewing employers and 
other stakeholders to explore NSCCS alignment with industry needs. 

Your individual responses will be kept confidential and aggregated for the report. No personally 
identifying information will be reported, and we will make every effort to protect your identity 
when we present our findings. Please review the Informed Consent document attached to this 
email prior to the interview. 

If you have any questions about the evaluation or your participation feel free to contact me, 

Susan Gallagher, or you may email Solutions IRB (our external review board charged with 
ensuring we treat evaluation study participants ethically) at participants@solutionsirb.com . 

Thank you for your participation, 

[SIGNATURE OF SENDER] 
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Interview Instructions 

ITEMS IN ITALICS SHOULD NOT BE READ TO INTER VIEWEE 


Phone Interview Introduction 

Hello, this is from Hezel Associates. iQn calling about the interview we 

have scheduled to discuss your involvement with the National STEM Consortium project at 
[college]. 

Is now still a convenient time to talk? 

As a reminder, your responses will be kept confidential and aggregated for the report. No 
personally identifying information will be reported, and we will make every effort to protect 
your identity when we present our findings. You can stop the interview at any time and skip any 
questions you are not comfortable answering. You can also choose to withdraw your responses. 

Have you read the informed consent document that was emailed to you? 

IF NOT, GO OVER THE MAJOR SECTIONS WITH THEM, ESPECIALLY BENEFITS AND 
RISKS. 

Do you have any questions concerning the consent form or the study? 

Do you agree to participate in the interview? 

I would like to record our interview to support my note-taking, and the recording will not be used 
for any other purpose. May I have your permission to record our conversation? 

IF PARTICIPANT DECLINES RECORDING, RESEARCHER WILL TAKE NOTES 
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Questions 

1. To clarify, you are involved with the [ technical track ] at [college], correct? 

( Interviewer : Use information in contact list) 

2. To start, please describe your involvement in the [ technical track ] at [college]. 

How did you get involved? 

What were you requested to do? 

How have you contributed? 

3. How do you see your relationship with the [ technical track] at [college] in the future? 5 
{Probe: evolution of technical con tent, hiring) 

4. What is your overall opinion on the [technical track] at [college]! 3 ' 4 ' 5 

5. How do you envision the [technical track] at [college] fitting into the future labor market in 
your region? 5 

What kind of impact do you think it will have on the labor market in your region? 5 

6. What types of positions would completers of the [technical track] qualify for at your 
company? 3 

{Probe: job titles, job level [entry, etc.]) 

What sort of salary ranges do these positions typically have? 

Describe the potential for employees in those positions to move up in the company, if 
any. 

7. Has your company hired any individuals (permanent or interns) who completed the 
[technical track ]? 3 

If no: 

Why not? 

What changes to the [technical track] would influence hiring, if any? 

If yes: 

What made that person (or those people) a good candidate for the position? 

What can you say about their performance on the job? 

{Probe: “soft skills” [e.g., professionalism, communication] and technical skills) 

8. What changes do you anticipate in terms of your company® future needs? 5 
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(Probe: technologies, industry standards, customer needs) 

Do you anticipate growth in the company? Please elaborate. 
Thank you, that 6, it for my questions, 

9. Is there anything else youQd like to say about the [ technical track]! 
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